←
vLLM1,542 posts
vLLM avatar✓

vLLM ✓

@vllm_project

A high-throughput and memory-efficient inference and serving engine for LLMs. Join slack.vllm.ai to discuss together with the community!

◷ Joined March 2024▧ 466 Photos and videos
vLLM avatar
vLLM@vllm_project·8h

🎉 Day-0 support for @TencentHunyuan’s Hy4 flash is here in vLLM! A 1M-token context and multimodal understanding for research, documents, slides and computer use. 224B total parameters, including 64B Engram, with 12B active. Hybrid attention, Latent MoE and Engram put efficiency at the center of the architecture. Run Hy4 flash on @NVIDIA GPUs with vLLM, with integration work by @inferact. Get started with the release image and deployment recipe: Docker image: vllm/vllm-openai:hy4-flash Recipe: recipes.vllm.ai/tencent/Hy4-…

Post attachment
0:30
vLLM avatar
vLLM@vllm_project·19h

Help shape the vLLM Semantic Router roadmap for Q4 2026–Q1 2027. Rate the features you need, add what’s missing, or propose a new work group. Your feedback sets priorities for upcoming releases. ~8 minutes. No sign-in. form.vllm-sr.ai/roadmap

vLLM avatar
vLLM@vllm_project·20h

More accepted tokens. Bounded draft KV memory. 🔥 The new DSpark for @Kimi_Moonshot’s Kimi K3.1 Pro, jointly developed by @Inferact and the @RedHat_AI Speculators team with vLLM + Speculators, brings: • Up to 30% higher acceptance length vs. the preview (HumanEval). • A 2K sliding window in every draft layer, keeping draft KV memory bounded even at 1M-token context. Efficient draft architectures need an inference stack that can serve them. vLLM’s redesigned KV management supports P/D disaggregation and different parallelism configurations, bringing sliding-window draft models into production. Comparison: DSpark k=7, temperature 1.0, top_p 0.95. Acceptance length includes the bonus token. Weights + vLLM config below. 👇

Post attachment
vLLM avatar
vLLM@vllm_project·20h

🎉 Day-0 support for AURA 2.0 in vLLM-Omni! A 9B streaming video model that watches the scene, starts relevant conversations, and acts on what it sees. 💬 Model-Initiated Conversation: decide when to speak and what to say, without waiting for a question. 🔧 Stream-Aware Tool Calling: use the live scene and conversation to call tools and bring results back into the dialogue. 🎯 Interactive Visual Grounding: point to the objects being discussed with frame-aligned boxes. vLLM-Omni powers the pipeline from speech recognition through video-language inference to speech generation. Session and prefix reuse, a fast path for silent turns, and sentence-level speech handoff deliver 344 ms median time to the first reply audio packet in the single-session setup with two ~140 GB accelerators. AURA 2.0 leads the four aggregate benchmark comparisons shown below: IPIBench, StreamArena, ProactiveVideoQA and PhoStream. Thanks to the AURA team at @Huawei Research, CUHK MMLab and HKUST! Project & demos: aurateam2026.github.io/Aura-… Weights on @huggingface: huggingface.co/aurateam/AURA…

Post attachment
vLLM avatar
vLLM@vllm_project·Nov 17

Great work from @sageai_official on Sage S2.1 Flash, now supported in vLLM! From molecular graphs to protein structures, it brings scientific data understanding together with retrieval, coding and tool use—checking intermediate results and refining the next step. Weights are available on @HuggingFace and @ModelScope2022.

Post attachmentPost attachmentPost attachment
vLLM avatar
vLLM@vllm_project·Nov 16

.@NVIDIA put together a video walkthrough and deployment guide for running GLM-5.3 NVFP4 locally on two DGX Stations with vLLM. The setup keeps all model weights in GPU memory, using tensor parallelism across two GB300 GPUs to serve requests through a single OpenAI-compatible API. It also enables MTP speculative decoding, FP8 KV cache, and prefix caching. Watch the walkthrough: youtube.com/watch?v=opRXPRnS… Get the deployment guide: developer.nvidia.com/blog/wh…

vLLM avatar
vLLM@vllm_project·Nov 16

The only thing faster than GB300 Dynamo + vLLM is Vera Rubin Dynamo + vLLM 🏎️💨. The first DeepSeek V4.1 Flash results on @NVIDIA Vera Rubin NVL72 are here, powered by Dynamo + vLLM on @SemiAnalysis_'s AgentX benchmark. Compared with GB300 NVL72 running Dynamo + vLLM, InferenceX's perf ruler shows: • 63.4× total token throughput per chip at 400 tok/s/user • 93.3× at 600 tok/s/user

Post attachment
vLLM avatar
⟳ vLLM retweeted
vLLM@vllm_project·Nov 12

We optimized DeepSeek-V4.1-Flash on vLLM for Vera Rubin NVL72 and reached 58× the throughput of GB300 NVL72 at 1.7 million tokens/s/GPU and 387 tokens/s/user on SemiAnalysis AgentX. Across the curve: 🔹 2.52 million tokens/s/GPU at peak throughput (2.7× GB300) 🔹 1.7 million tokens/s/GPU at P90 387 tokens/s/user (58× GB300) 🔹 1,810 tokens/s at one concurrent session with TP 1/4

Post attachment
vLLM avatar
vLLM@vllm_project·Nov 12

Rubin computes fast enough that a rack serves far more sessions at once, and their live prefixes outgrow any per-rank cache, so we pooled the rack’s memory: one Mooncake KV pool across the DRAM and spare HBM of every tray, and the Engram tables sharded across GPU HBM and read over NVLink. Even at very high concurrency the hit rate stays at 96.5–97%, close to the theoretical maximum. With no session pinned to a rank, Dynamo’s router sends each request to the least-loaded one: TTFT P90 drops from 0.98 s to 0.38 s. 3/4

Post attachment
vLLM avatar
vLLM@vllm_project·Nov 12

We reworked the MoE, attention, GEMM, mHC and indexer kernels for Rubin: bandwidth-bound data stays in each SM's own memory domain, SM107's wider MMA instructions do more work per issue. DEP2 prefill steps are about 28% faster, and DEP decode steps about 25% faster. We also ported our specialized decode path to Rubin and rebuilt most of it for SM107. A step there is a chain of small kernels near their latency floor, so the wins are the gaps: all-reduces fused into the kernels behind them, the shared expert inside the routed GEMMs, and loads that start before the wait. 2/4

vLLM avatar
vLLM@vllm_project·Nov 12

🎉 Day-0 support for @thinkymachines’ Inkling-2-Small in vLLM! A 500B MoE with text + image inputs and text output. Bring the image into the conversation: ask a question with visual context, and get a text response from the same model. Under the hood: a new ViT image encoder, mixed NVFP4/MXFP8 MoE layers, and DSpark speculative decoding. @inferact leads the vLLM integration and optimization, working with Thinking Machines Lab on model code and checkpoints. The work spans MoE and attention kernels, the vision encoder and image preprocessing, plus draft-model GEMMs for speculative decoding. Thanks to the TML, Inferact and vLLM teams bringing the model to an open inference stack!

vLLM avatar
vLLM@vllm_project·Nov 12

vllm serve thinkingmachines/Inkling-2-Small --tensor-parallel-size 8 \ --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' Recipe: recipes.vllm.ai/thinkingmach… Thanks to the Thinking Machines team 🙏