Exciting thing here (really exciting!). Introducing our new Nsight Perf SDK for AI, a beta preview, that enables a very, very low overhead collection technology at profiling data rates aka 10Khz+ with the ability to attach to running workloads, see your CUDA kernel traces and tie those to hardware metrics...
Check it out, give feedback and yes we are building other fun stuff on top of this... stay tuned!
github.com/NVIDIA/nsight-per…
3/5
@NVIDIAAI Dynamo's AIPerf benchmarking tool takes things a step further with the following metric comparisons:
- Throughput vs Interactivity
- TTFT Latency vs Throughput
- Inter-token Latency vs Throughput
The 1st image is a cleaner up to 8 concurrent requests chart, with the 2nd image showing up to 32 concurrent requests where a drop in median TTFT occurred from 8-16 concurrent requests
Super excited to share an update to my blog from a few months ago on how we’re optimizing dynamo and the inference frameworks for agentic inference performance. This is a problem that we’ve been thinking a lot about and many teams have already put these optimizations and more into production!
Agentic workloads generate long, expanding context across repeated model calls, tool execution, and parallel subagents. Much of this context remains stored in the KV cache while an agent waits on external tools or pending operations.
Efficient serving relies on preserving and reusing this PyTorch tensor state while maintaining sub-second latency targets across concurrent sessions.
Read our latest blog from @nvidia to learn how NVIDIA Dynamo integrates its router, inference engine, and KV-cache manager to deliver seamless end-to-end serving for agentic workloads: bit.ly/4emlt21
Dynamo 2.0 aspires to be the best post-training inference system and this is just the first step.. Lots of exciting work in progress. Check the Dynamo GitHub for more PRs 😁
Super excited to share something we’ve been working on:
Coreweave Forge’s RL rollouts run on Dynamo!
We’ve been doing incredible work to make Dynamo a SOTA rollout backend! Check out the perf numbers in the blog post!
Our RL stack now supports NIXL weight transfer, reducing trainer-to-inference transfer time 9x compared with NCCL: from 86 seconds down to single-digit seconds for an 800B-parameter model, and even <4 seconds in our experiments.
For prime-rl users, this means over 25% more throughput end-to-end compared with our previous speed. It also clears the way for fault-tolerant, elastic inference scaling that NCCL's rigid process groups made difficult.
Join @baseten, @nvidia Dynamo, and SGLang on 9/10 for an evening dedicated to the infrastructure behind reinforcement learning post-training!
We will present the latest on SGLang & Miles. Miles is an RL training framework using SGLang as the rollout inference engine to make RL post-training fast and reliable. Come meet us in person!
🕐 Thursday, September 10, 6:00–9:00 PM
👉 Register now
luma.com/BaseDynSGL
Super happy to share our intention to join forces with NVIDIA in a $12,930,300,000 acquisition 💛💚
10 years after starting Hugging Face, open-source AI is at an inflection point. Thanks to the community, we’ve shown that it can be a complement, and even an alternative, to closed-source APIs. But for it to happen at larger scale, it needs more compute, more support, more collaboration and more visibility. That’s why we went to talk to Jensen, who offered to do exactly that with us.
In addition to doubling down on NVIDIA’s massive contributions to open-source AI (I called them the “King of American open-source AI” earlier this year), they’ve committed to strongly supporting Hugging Face and our mission while keeping the platform open, independent and compute agnostic. The founders and the team are all staying to keep pushing this mission forward.
Together, we think we can make open source the default way to build AI, with the goal of empowering 100 million AI builders to own their intelligence rather than rent it.
Excited about the next 10 years! 🤗🤗🤗
Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not.
NVIDIA’s open-source srt-slurm (github.com/NVIDIA/srt-slurm) provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services.
It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm.
Great work to @0xishand and the rest of the @NVIDIAAI team on this!
Delivery day at our Microsoft DCs as the first production Vera Rubins arrive. A huge thank you to our partners at @nvidia and our Azure hardware and datacenter teams for all the incredible work that brought us to this milestone!
Qwen 3.8-2.4T recipes for Dynamo, straight out of the oven for Day 0!
We have SGLang and VLLM recipes for chat/agentic use cases!
Let us know what you think!
github.com/ai-dynamo/dynamo/…
For those who know me, today is a special milestone, seeing years of our research become @nvidia products. Today at FMS, we are excited to make the following major announcements.
(1/6)
Introducing Dynamo Snapshot, our approach for fast startup for inference workloads on Kubernetes, which reduces startup time from minutes to under 5 seconds.
In production inference deployments demand fluctuates over time. Cold-starting inference workloads can take minutes, leaving idle GPUs that generate no tokens and serve no requests.
Snapshot leverages GMS to enable concurrent weight restoration over a high-speed interconnect, while using Linux native AIO and parallel memfd restoration to accelerate CRIU restore performance.
vLLM's PegaFlow and Dynamo's KVBM are converging on the same bet: external KV cache as a standalone Rust service over a connector boundary. The interesting design choice between them - does the inference engine own the prefix index, or does the storage layer?
A vLLM MoE deployment's DP/EP topology used to be locked in at launch — scaling or swapping config meant a full restart, in-flight traffic dropped. Elastic Expert Parallelism changes that. One API call resizes a live deployment:
curl -X POST localhost:8000/scale_elastic_ep \
-d '{"new_data_parallel_size": 16}'
Under the hood: standby comm groups span the target topology, EPLB redistributes experts across the new EP group, and weights are transferred directly between GPUs over NVIDIA NVLink/RDMA. The same runtime reconfiguration path is what fault-tolerant serving needs: evict failed ranks, redistribute their experts, bring replacements back, no restart.
Thanks to @NVIDIAAI, Sky Computing, @anyscalecompute, @RedHat_AI, and the community.
📖 vllm.ai/blog/2026-05-14-elas…