Pipeline parallelism (PP) is rarely picked for prefill because of its pipeline bubbles.
We think agentic serving changes that. On @MiniMax_AI MiniMax-M3.1-Flash with @vllm_project on two NVIDIA B300s, PP2 sustained 23% more requests per GPU than DEP2 and 17% more than TP2 at a 2 s p50 TTFT target.
1/5
Thanks for the shoutout @digitalocean!
Our own @yifandotqiao breaks down the cache work behind taking @Kimi_Moonshot K3 from 84K to 480K tokens/min per card with @vllm_project.
Serving multi-turn agents well starts with making their long input histories cheaper to process.
A typical agentic session runs 43 turns with 142K tokens in and 444 tokens out. Serving that well comes down to the cache.
@yifandotqiao of @Inferact (the team behind vLLM) on taking one card from 84K to 480K tokens a minute on @Kimi_Moonshot K3. do.co/4zJtvdv
We contribute this research to vLLM, and because the Inferact platform runs on vLLM, the same improvements reach our production serving.
inferact.ai/news/pipeline-pa…
5/5
PP still pays for pipeline bubbles, the time one stage waits for the other.
We moved the stage hand-off from NCCL to CUDA copy engines, so stage 0 pushes its batch and moves on without taking SMs from compute. That cut PP2’s p50 TTFT by 8–16% at the same load.
Even so, TP2 still leads at light load and at a 1 s target, so the best layout depends on your TTFT target.
4/5
With latency covered, what remains is throughput, which depends on how often the GPUs wait for each other.
TP2 all-reduces twice in every layer, taking 18% of its GPU time. DEP2’s ranks meet in every MoE layer, so a 4:2 request split cost it 15%. PP2 hands off once per batch.
3/5
An agent turn adds a few thousand tokens to a long, mostly cached context, then spends seconds decoding and calling tools, so a few seconds of TTFT is enough.
We tested a 100K-token cached prefix plus 7.5K–12.5K new tokens. PP2’s light-load TTFT was 0.45 s against TP2’s 0.31 s.
2/5
Our CEO & cofounder @simon_mo_ is speaking at #InferCon on November 16 in San Francisco, convened by #Datadog!
Building AI systems? Join us to explore inference, infrastructure, and operating AI at scale.
Apply to attend: infercon.ai
Our team led the @vllm_project work behind this result: 58× the throughput of GB300 NVL72 on @SemiAnalysis_ AgentX at at 1.7 million tokens/s/GPU at 387 tokens/s per user.
Our view is that new hardware only pays off when the serving stack is rebuilt around it. Reaching this result on Vera Rubin took work at three levels:
- Kernels: we rebuilt DeepSeek’s megakernels for Rubin’s split memory domains and SM107’s wider MMA instructions.
- Low latency: we run a specialized decode path rebuilt for Rubin, reading weights from each SM’s own memory domain, fusing its kernels into each other and starting their loads early.
- KV Caching: we pooled the KV cache and Engram tables across the rack’s memory, and Dynamo routes each request to the least-loaded prefill rank.
We build these optimizations in open-source vLLM and run them in production for our customers.
Learn more at inferact.ai
We optimized DeepSeek-V4.1-Flash on vLLM for Vera Rubin NVL72 and reached 58× the throughput of GB300 NVL72 at 1.7 million tokens/s/GPU and 387 tokens/s/user on SemiAnalysis AgentX.
Across the curve:
🔹 2.52 million tokens/s/GPU at peak throughput (2.7× GB300)
🔹 1.7 million tokens/s/GPU at P90 387 tokens/s/user (58× GB300)
🔹 1,810 tokens/s at one concurrent session with TP
1/4
@TencentHunyuan and Inferact are partnering on Tencent Hy4, Tencent’s newest flagship model.
Hy4, the hosted model behind Tencent’s API, is already live on the Inferact Platform and carrying production traffic. Tencent sets the checkpoint and the quality bar. We validate the serving path against it before taking traffic, then operate the deployment end to end and keep tuning it as workloads change.
Hy4 Preview is the open-weight release of the same model, and the two teams are co-leading its native @vllm_project support. Tencent defines the architecture and reference behavior. Inferact builds and hardens the engine support, and ships the recipes to get from checkpoint download to a working serving system.
Production feeds open source, and open source feeds production. Our partnership is built on this loop.
Thank you to the Tencent HY team. Full announcement:
inferact.ai/news/tencent-par…
Inferact now serves more than 7 trillion tokens a day.
That traffic runs on the Inferact Platform, which serves frontier models with optimized vLLM across heterogeneous hardware and diverse workloads. For context, OpenRouter averaged 19 trillion tokens a day of open-weight traffic over the past 30 days, across all of its providers.
The platform improves through a data flywheel. Production traffic shows us where to improve vLLM, partnerships with accelerators like NVIDIA, AMD, and Google bring those improvements to new hardware, and partnerships with model labs like MiniMax and Kimi bring them to new models from launch day.
Next, we’re automating that flywheel with agents that find and validate optimizations on their own, so the platform improves with every token it serves.
inferact.ai/news/7-trillion-…
Today, @Kimi_Moonshot launched Kimi K3.1 Pro, and we are announcing an expanded partnership with Moonshot AI to bring the Kimi model series to production with vLLM.
Kimi K3.1 Pro is available on the Inferact Platform from launch. We held it to the same standard as Kimi K3, which we brought onto the platform after completing final qualification and Kimi Vendor Verification.
Inferact also partners with Moonshot AI on @vllm_project support for each Kimi release. For Kimi K3.1 Pro, developers can serve the model in vLLM today.
Our work with Moonshot AI on each model starts before release. We tune and validate the model serving path together, then ship improvements to the community, such as a speculative decoding model that raises generation throughput. What we learn serving Kimi in production is upstreamed to vLLM.
Thank you to the Moonshot AI team for the partnership. Read the full announcement:
inferact.ai/news/moonshot-pa…
Our co-founder and CEO @simon_mo_ joined @Modal’s Runtime event to discuss what it takes to serve agentic AI in production and what’s next for vLLM.
Watch the full talk: youtube.com/watch?v=dh36XYa-…
Our teammate Ziming Huang joined Ascend's inference meetup on October 22 to share how we take @Kimi_Moonshot's Kimi K3 from day-0 support to production serving.
Early model access, architecture understanding, and the runtime and kernel work behind KDA, MLA and LatentMoE—the engineering behind making a model ready to serve.
Thanks to Huawei and the Ascend community for having us!
The same engineering runs on the Inferact platform. We serve leading open models like DeepSeek-V4.1-Flash on your hardware or ours, tuned for your workload, with the vLLM maintainers on the other end.
inferact.ai
DeepSeek open-sourced DeepInfer this week, the engine behind its own DeepSeek-V4.1-Flash serving. We think it is the most useful kind of release: a purpose-built engine, originally built on @vllm_project, that shows what one model can do on one piece of hardware.
Our team got vLLM to match it on DeepSeek's own decode configuration, using half the GPUs, and then kept going to where an offline reference engine does not reach: online serving across a full GB300 NVL72 rack, at 954K tokens/s per GPU on SemiAnalysis AgentX and $38.96 of token value per GPU-hour.
Our view is that the gap between a lab’s reference engine and the engine you run in production should be close to zero, and that closing it is mostly careful systems work rather than new algorithms.
vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64.
DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.