
The big theme at @KubeCon_ this week was inference. And "Inference" is such a loaded term, so let's think about each piece and where it matters for production teams. 1. GPUs and memory: Making sure you have enough, but at the same time, how it's allocated and where/how it can be shared. Dynamic Resource Allocation (DRA) was talked about a lot here. It handles the GPU slicing, and KEDA/Karpenter handle the scaling. 2. vLLM to serve LLMs and help with KV caching via PagedAttention. The goal is to cache memory efficiently, which should reduce memory waste and enhance throughput. 3. Inference-aware Gateways: AI Gateways with the Gateway API Inference extension capabilities, like @agentgateway, for routing to LLMs. 4. Distributed inference/scheduling: llm-d, which sits between vLLM for cache-aware request scheduling. 5. Isolation: Agents running on this infrastructure need a way to be truly sandboxed, with controls around ingress and egress. The sandbox layer needs to come from a lower-level engine that runs efficiently and helps implement not only isolation, but scale. @kagent_dev + Agent Substrate. 6. Where it all runs: @kubernetesio This six-layer stack is inference infrastructure and agent infrastructure to serve LLMs and Agents efficiently. #agenticai #ai #kubernetes










