←
Senthilkumar Gopal1,244 posts
Senthilkumar Gopal avatar

Senthilkumar Gopal

@sengopal

Engineering @ NVIDIA Dynamo. Planet scale LLM Inference systems - Disagg Serving, Tiered KV, . ex-AWS|Trainium. @GeorgiaTech. Opinions my own.

◷ Joined May 2009▧ 57 Photos and videos
Tony Mongkolsmai avatar
⟳ Senthilkumar Gopal retweeted
Tony Mongkolsmai@tonymongkolsmai·Nov 5

Exciting thing here (really exciting!). Introducing our new Nsight Perf SDK for AI, a beta preview, that enables a very, very low overhead collection technology at profiling data rates aka 10Khz+ with the ability to attach to running workloads, see your CUDA kernel traces and tie those to hardware metrics... Check it out, give feedback and yes we are building other fun stuff on top of this... stay tuned! github.com/NVIDIA/nsight-per…

Yannick Monye avatar
⟳ Senthilkumar Gopal retweeted
Yannick Monye@keennay·Nov 2

3/5 @NVIDIAAI Dynamo's AIPerf benchmarking tool takes things a step further with the following metric comparisons: - Throughput vs Interactivity - TTFT Latency vs Throughput - Inter-token Latency vs Throughput The 1st image is a cleaner up to 8 concurrent requests chart, with the 2nd image showing up to 32 concurrent requests where a drop in median TTFT occurred from 8-16 concurrent requests

Post attachmentPost attachment
ishan avatar
⟳ Senthilkumar Gopal retweeted
ishan@0xishand·Oct 9

Super excited to share an update to my blog from a few months ago on how we’re optimizing dynamo and the inference frameworks for agentic inference performance. This is a problem that we’ve been thinking a lot about and many teams have already put these optimizations and more into production!

Agentic workloads generate long, expanding context across repeated model calls, tool execution, and parallel subagents. Much of this context remains stored in the KV cache while an agent waits on external tools or pending operations. Efficient serving relies on preserving and reusing this PyTorch tensor state while maintaining sub-second latency targets across concurrent sessions. Read our latest blog from @nvidia to learn how NVIDIA Dynamo integrates its router, inference engine, and KV-cache manager to deliver seamless end-to-end serving for agentic workloads: bit.ly/4emlt21

Senthilkumar Gopal avatar
Senthilkumar Gopal@sengopal·Oct 2

Dynamo 2.0 aspires to be the best post-training inference system and this is just the first step.. Lots of exciting work in progress. Check the Dynamo GitHub for more PRs 😁

Super excited to share something we’ve been working on: Coreweave Forge’s RL rollouts run on Dynamo! We’ve been doing incredible work to make Dynamo a SOTA rollout backend! Check out the perf numbers in the blog post!

Senthilkumar Gopal avatar
Senthilkumar Gopal@sengopal·Sep 4

Just Dynamo things 😊

Our RL stack now supports NIXL weight transfer, reducing trainer-to-inference transfer time 9x compared with NCCL: from 86 seconds down to single-digit seconds for an 800B-parameter model, and even <4 seconds in our experiments. For prime-rl users, this means over 25% more throughput end-to-end compared with our previous speed. It also clears the way for fault-tolerant, elastic inference scaling that NCCL's rigid process groups made difficult.

Post attachment
SGLang avatar
⟳ Senthilkumar Gopal retweeted
SGLang@sgl_project·Sep 2

Join @baseten, @nvidia Dynamo, and SGLang on 9/10 for an evening dedicated to the infrastructure behind reinforcement learning post-training! We will present the latest on SGLang & Miles. Miles is an RL training framework using SGLang as the rollout inference engine to make RL post-training fast and reliable. Come meet us in person! 🕐 Thursday, September 10, 6:00–9:00 PM 👉 Register now luma.com/BaseDynSGL

clem 🤗 avatar
⟳ Senthilkumar Gopal retweeted
clem 🤗@ClementDelangue·Sep 3

Super happy to share our intention to join forces with NVIDIA in a $12,930,300,000 acquisition 💛💚 10 years after starting Hugging Face, open-source AI is at an inflection point. Thanks to the community, we’ve shown that it can be a complement, and even an alternative, to closed-source APIs. But for it to happen at larger scale, it needs more compute, more support, more collaboration and more visibility. That’s why we went to talk to Jensen, who offered to do exactly that with us. In addition to doubling down on NVIDIA’s massive contributions to open-source AI (I called them the “King of American open-source AI” earlier this year), they’ve committed to strongly supporting Hugging Face and our mission while keeping the platform open, independent and compute agnostic. The founders and the team are all staying to keep pushing this mission forward. Together, we think we can make open source the default way to build AI, with the goal of empowering 100 million AI builders to own their intelligence rather than rent it. Excited about the next 10 years! 🤗🤗🤗

Post attachment
SemiAnalysis avatar
⟳ Senthilkumar Gopal retweeted
SemiAnalysis@SemiAnalysis_·Aug 27

Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not. NVIDIA’s open-source srt-slurm (github.com/NVIDIA/srt-slurm) provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services. It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm. Great work to @0xishand and the rest of the @NVIDIAAI team on this!

Post attachment
Satya Nadella avatar
⟳ Senthilkumar Gopal retweeted
Satya Nadella@satyanadella·Aug 21

Delivery day at our Microsoft DCs as the first production Vera Rubins arrive. A huge thank you to our partners at @nvidia and our Azure hardware and datacenter teams for all the incredible work that brought us to this milestone!

Post attachmentPost attachmentPost attachment
Vikram avatar
⟳ Senthilkumar Gopal retweeted
Vikram@msharmavikram·Aug 4

For those who know me, today is a special milestone, seeing years of our research become @nvidia products. Today at FMS, we are excited to make the following major announcements. (1/6)

NVIDIA avatar
⟳ Senthilkumar Gopal retweeted
NVIDIA@nvidia·May 29

A new era of PC. 25.0528, 121.5990

NVIDIA AI avatar
⟳ Senthilkumar Gopal retweeted
NVIDIA AI@NVIDIAAI·May 27

Introducing Dynamo Snapshot, our approach for fast startup for inference workloads on Kubernetes, which reduces startup time from minutes to under 5 seconds. In production inference deployments demand fluctuates over time. Cold-starting inference workloads can take minutes, leaving idle GPUs that generate no tokens and serve no requests. Snapshot leverages GMS to enable concurrent weight restoration over a high-speed interconnect, while using Linux native AIO and parallel memfd restoration to accelerate CRIU restore performance.

Post attachment
Senthilkumar Gopal avatar
Senthilkumar Gopal@sengopal·May 26

vLLM's PegaFlow and Dynamo's KVBM are converging on the same bet: external KV cache as a standalone Rust service over a connector boundary. The interesting design choice between them - does the inference engine own the prefix index, or does the storage layer?

Senthilkumar Gopal avatar
Senthilkumar Gopal@sengopal·May 22

And this will work with Nvidia Dynamo seamlessly 😊

A vLLM MoE deployment's DP/EP topology used to be locked in at launch — scaling or swapping config meant a full restart, in-flight traffic dropped. Elastic Expert Parallelism changes that. One API call resizes a live deployment: curl -X POST localhost:8000/scale_elastic_ep \ -d '{"new_data_parallel_size": 16}' Under the hood: standby comm groups span the target topology, EPLB redistributes experts across the new EP group, and weights are transferred directly between GPUs over NVIDIA NVLink/RDMA. The same runtime reconfiguration path is what fault-tolerant serving needs: evict failed ranks, redistribute their experts, bring replacements back, no restart. Thanks to @NVIDIAAI, Sky Computing, @anyscalecompute, @RedHat_AI, and the community. 📖 vllm.ai/blog/2026-05-14-elas…

Post attachment