←
Michael Levan👨🏻‍💻☕️10,938 posts
Michael Levan👨🏻‍💻☕️ avatar

Michael Levan👨🏻‍💻☕️

@TheNJDevOpsGuy

Building How AI Runs in Prod | Principal AI Architect & Forward Deployed Engineer | AAIF & CNCF Ambassador | Microsoft MVP | 4x Author & International Speaker

◷ Joined September 2017▧ 2,565 Photos and videos
Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 14

The big theme at @KubeCon_ this week was inference. And "Inference" is such a loaded term, so let's think about each piece and where it matters for production teams. 1. GPUs and memory: Making sure you have enough, but at the same time, how it's allocated and where/how it can be shared. Dynamic Resource Allocation (DRA) was talked about a lot here. It handles the GPU slicing, and KEDA/Karpenter handle the scaling. 2. vLLM to serve LLMs and help with KV caching via PagedAttention. The goal is to cache memory efficiently, which should reduce memory waste and enhance throughput. 3. Inference-aware Gateways: AI Gateways with the Gateway API Inference extension capabilities, like @agentgateway, for routing to LLMs. 4. Distributed inference/scheduling: llm-d, which sits between vLLM for cache-aware request scheduling. 5. Isolation: Agents running on this infrastructure need a way to be truly sandboxed, with controls around ingress and egress. The sandbox layer needs to come from a lower-level engine that runs efficiently and helps implement not only isolation, but scale. @kagent_dev + Agent Substrate. 6. Where it all runs: @kubernetesio This six-layer stack is inference infrastructure and agent infrastructure to serve LLMs and Agents efficiently. #agenticai #ai #kubernetes

Post attachment
Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·7h

My 5th book is officially published! 🎉 AI is moving so fast in every aspect of our lives from work to personal to healthcare to daily tasks... And especially for everyone in the technical space. That's why where you run agentic workloads and infrastructure is the make or break between a successful implementation of AI in production and just testing. That's what this book covers 🤓 link.springer.com/book/10.10… #agenticai #ai #kubernetes

Post attachment
Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 17

Reinforcement Learning (RL) in a nutshell: 1. Have an Agent perform multiple tasks. 2. Learn from those tasks 3. Update the weights in the LLMs that the Agent used after the tasks are done with what it learned. And this is paramount for post-training. The reason why is because post-training is the "quality" that you see in an LLM. It's not about the info it was trained on, it's about how that info can be portrayed to the user. And where/how you're running RL comes down to sandboxing the post-training process and properly testing the output to ensure it's what you expect. In this blog post, I break down how to do it with Agent Substrate, vLLM, and a NVIDIA H100. cloudnativedeepdive.com/open… #agenticai #ai #kubernetes

Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 14

When people think about Agent Substrate, sandboxing/isolation and security come to mind. However, there’s another huge piece out of the box. Efficiently running Agents. I’m not even talking about “there are hundreds of Agents running”. Even if you have 2 or 5 Agents running, you still get a maximum benefit of running Agents inside of Actors vs across multiple Pods. - Agents are only up when they’re interacted with. - You get a central place to store context/convo/memory. - Agents can be scaled across clusters. - Multiple Agents, one Pod. The performance and ability to cut down resource usage can be seen and felt for agentic workloads big and small. #agenticai #ai #kubernetes

Post attachment
Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 14

Reinforcement Learning (RL) on Agent Substrate; TLDR. An Actor isn't where the training runs; it's where the environment runs. The Actor executes what the trainer created. The trainer builds the test program and the Actor runs it. Training happens entirely outside of Substrate. Agent Substrate's primary job is "run this untrusted Python" in a secure, isolated, and sandboxed environment. #agenticai #ai #kubernetes

Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 8

This used to bug me big time (@claudeai adding a bunch of comments). Clean code best practices are "of course add comments, but not the comments you don't need". The reason why we never added a crap ton of comments in a codebase before is because it would drastically slow down humans because they had to read through it all. It's not like comments slow the runtime/compilation down though, so... In the world of agentic programming/coding, I'm beginning to think Claude does it so when Claude look at the codebase a week from now, a month from now, a year from now, it has reference points. #agenticai #ai #kubernetes

Post attachment
Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 7

Labs have been updated for Agent Substrate v1.0.0 #agenticai #ai #kubernetes

The Agentic Field guide currently has 16 hands-on labs for sandboxing and isolation. Two sections: ✅ Isolated Environments With Agent Substrate ✅ Isolated Agents With Kagent + Agent Substrate This field guide will be updated weekly, so if you're looking to learn more about agentic sandboxing, isolation, governance, and how it works with Agent Substrate, this is the place for you. agenticfieldguide.ai/ #agenticai #ai #kubernetes

Michael Levan👨🏻‍💻☕️ avatar
Michael Levan👨🏻‍💻☕️@TheNJDevOpsGuy·Nov 6

Agent Substrate v1.0.0 is officially out. There's A LOT to a GA release of any software, so I wanted to map out the 5 top-of-mind things. 1. Rolling upgrades are supported 2. One actor's memory use can no longer take down its worker 3. Suspend and resume recover from more failures 4. Egress and router fixes 5. Easier to get started and connect clients #agenticai #ai #kubernetes