←
Inferact183 posts
Inferact avatar

Inferact

@inferact

Building the future of inference through @vllm_project

◷ Joined December 2025▧ 26 Photos and videos
Inferact avatar
⟳ Inferact retweeted
Inferact@inferact·Nov 16

Pipeline parallelism (PP) is rarely picked for prefill because of its pipeline bubbles. We think agentic serving changes that. On @MiniMax_AI MiniMax-M3.1-Flash with @vllm_project on two NVIDIA B300s, PP2 sustained 23% more requests per GPU than DEP2 and 17% more than TP2 at a 2 s p50 TTFT target. 1/5

Post attachment
Inferact avatar
Inferact@inferact·19h

Thanks for the shoutout @digitalocean! Our own @yifandotqiao breaks down the cache work behind taking @Kimi_Moonshot K3 from 84K to 480K tokens/min per card with @vllm_project. Serving multi-turn agents well starts with making their long input histories cheaper to process.

A typical agentic session runs 43 turns with 142K tokens in and 444 tokens out. Serving that well comes down to the cache. @yifandotqiao of @Inferact (the team behind vLLM) on taking one card from 84K to 480K tokens a minute on @Kimi_Moonshot K3. do.co/4zJtvdv

0:47
Inferact avatar
Inferact@inferact·Nov 16

PP still pays for pipeline bubbles, the time one stage waits for the other. We moved the stage hand-off from NCCL to CUDA copy engines, so stage 0 pushes its batch and moves on without taking SMs from compute. That cut PP2’s p50 TTFT by 8–16% at the same load. Even so, TP2 still leads at light load and at a 1 s target, so the best layout depends on your TTFT target. 4/5

Post attachment
Inferact avatar
Inferact@inferact·Nov 16

With latency covered, what remains is throughput, which depends on how often the GPUs wait for each other. TP2 all-reduces twice in every layer, taking 18% of its GPU time. DEP2’s ranks meet in every MoE layer, so a 4:2 request split cost it 15%. PP2 hands off once per batch. 3/5

Post attachment
Inferact avatar
Inferact@inferact·Nov 16

An agent turn adds a few thousand tokens to a long, mostly cached context, then spends seconds decoding and calling tools, so a few seconds of TTFT is enough. We tested a 100K-token cached prefix plus 7.5K–12.5K new tokens. PP2’s light-load TTFT was 0.45 s against TP2’s 0.31 s. 2/5

Post attachment
Inferact avatar
Inferact@inferact·Nov 12

Our team led the @vllm_project work behind this result: 58× the throughput of GB300 NVL72 on @SemiAnalysis_ AgentX at at 1.7 million tokens/s/GPU at 387 tokens/s per user. Our view is that new hardware only pays off when the serving stack is rebuilt around it. Reaching this result on Vera Rubin took work at three levels: - Kernels: we rebuilt DeepSeek’s megakernels for Rubin’s split memory domains and SM107’s wider MMA instructions. - Low latency: we run a specialized decode path rebuilt for Rubin, reading weights from each SM’s own memory domain, fusing its kernels into each other and starting their loads early. - KV Caching: we pooled the KV cache and Engram tables across the rack’s memory, and Dynamo routes each request to the least-loaded prefill rank. We build these optimizations in open-source vLLM and run them in production for our customers. Learn more at inferact.ai

We optimized DeepSeek-V4.1-Flash on vLLM for Vera Rubin NVL72 and reached 58× the throughput of GB300 NVL72 at 1.7 million tokens/s/GPU and 387 tokens/s/user on SemiAnalysis AgentX. Across the curve: 🔹 2.52 million tokens/s/GPU at peak throughput (2.7× GB300) 🔹 1.7 million tokens/s/GPU at P90 387 tokens/s/user (58× GB300) 🔹 1,810 tokens/s at one concurrent session with TP 1/4

Post attachment
Inferact avatar
⟳ Inferact retweeted
Inferact@inferact·Nov 11

@TencentHunyuan and Inferact are partnering on Tencent Hy4, Tencent’s newest flagship model. Hy4, the hosted model behind Tencent’s API, is already live on the Inferact Platform and carrying production traffic. Tencent sets the checkpoint and the quality bar. We validate the serving path against it before taking traffic, then operate the deployment end to end and keep tuning it as workloads change. Hy4 Preview is the open-weight release of the same model, and the two teams are co-leading its native @vllm_project support. Tencent defines the architecture and reference behavior. Inferact builds and hardens the engine support, and ships the recipes to get from checkpoint download to a working serving system. Production feeds open source, and open source feeds production. Our partnership is built on this loop. Thank you to the Tencent HY team. Full announcement: inferact.ai/news/tencent-par…

Inferact avatar
Inferact@inferact·Nov 5

Inferact now serves more than 7 trillion tokens a day. That traffic runs on the Inferact Platform, which serves frontier models with optimized vLLM across heterogeneous hardware and diverse workloads. For context, OpenRouter averaged 19 trillion tokens a day of open-weight traffic over the past 30 days, across all of its providers. The platform improves through a data flywheel. Production traffic shows us where to improve vLLM, partnerships with accelerators like NVIDIA, AMD, and Google bring those improvements to new hardware, and partnerships with model labs like MiniMax and Kimi bring them to new models from launch day. Next, we’re automating that flywheel with agents that find and validate optimizations on their own, so the platform improves with every token it serves. inferact.ai/news/7-trillion-…

Inferact avatar
Inferact@inferact·Nov 4

Today, @Kimi_Moonshot launched Kimi K3.1 Pro, and we are announcing an expanded partnership with Moonshot AI to bring the Kimi model series to production with vLLM. Kimi K3.1 Pro is available on the Inferact Platform from launch. We held it to the same standard as Kimi K3, which we brought onto the platform after completing final qualification and Kimi Vendor Verification. Inferact also partners with Moonshot AI on @vllm_project support for each Kimi release. For Kimi K3.1 Pro, developers can serve the model in vLLM today. Our work with Moonshot AI on each model starts before release. We tune and validate the model serving path together, then ship improvements to the community, such as a speculative decoding model that raises generation throughput. What we learn serving Kimi in production is upstreamed to vLLM. Thank you to the Moonshot AI team for the partnership. Read the full announcement: inferact.ai/news/moonshot-pa…

Inferact avatar
Inferact@inferact·Oct 29

Our teammate Ziming Huang joined Ascend's inference meetup on October 22 to share how we take @Kimi_Moonshot's Kimi K3 from day-0 support to production serving. Early model access, architecture understanding, and the runtime and kernel work behind KDA, MLA and LatentMoE—the engineering behind making a model ready to serve. Thanks to Huawei and the Ascend community for having us!

Post attachment
Inferact avatar
Inferact@inferact·Oct 24

DeepSeek open-sourced DeepInfer this week, the engine behind its own DeepSeek-V4.1-Flash serving. We think it is the most useful kind of release: a purpose-built engine, originally built on @vllm_project, that shows what one model can do on one piece of hardware. Our team got vLLM to match it on DeepSeek's own decode configuration, using half the GPUs, and then kept going to where an offline reference engine does not reach: online serving across a full GB300 NVL72 rack, at 954K tokens/s per GPU on SemiAnalysis AgentX and $38.96 of token value per GPU-hour. Our view is that the gap between a lab’s reference engine and the engine you run in production should be close to zero, and that closing it is mostly careful systems work rather than new algorithms.

vLLM matches DeepInfer, DeepSeek's own engine for DeepSeek-V4.1-Flash, on DeepSeek's decode configuration across batch sizes 8 to 256, and does it at DEP32 against DeepInfer’s EP64DP64. DeepInfer was originally built on vLLM. By integrating DeepSeek's open-source kernels and extensively optimizing prefill and decode, vLLM now matches DeepInfer's performance curves.

Post attachment