←
Simon Mo686 posts
Simon Mo avatar✓

Simon Mo ✓

@simon_mo_

building @inferact for @vllm_project

◷ Joined July 2018▧ 32 Photos and videos
Simon Mo avatar
Simon Mo@simon_mo_·Jan 22

vLLM has grown to 2000+ contributors scale with a diverse community of model, hardwares, and applications. I see @vllm_project on the path of becoming the world's inference engine and @inferact to accelerate AI progress. We cannot be more excited about the road ahead.

Post attachment
SemiAnalysis avatar
⟳ Simon Mo retweeted
SemiAnalysis@SemiAnalysis_·21h

ALERT🚨: In less than 20 days, AMD MI355X @vllm_project performance on MiniMax M3.1 Flash has increased by 4x! This is achieved using the new @Inferact AMD kernels written in FlyDSL, which are deterministic, meaning they never use atomic adds to accumulate their outputs. Great work by @vllm_project, @Inferact, and @AnushElangovan on this amazing achievement!

Post attachment
Woosuk Kwon avatar
⟳ Simon Mo retweeted
Woosuk Kwon@woosuk_k·Nov 17

true. Real-world performance is nonlinear to FLOPS, HBM bandwidth, and model size. With the right optimizations, we can make it superlinear

Yifan Qiao avatar
⟳ Simon Mo retweeted
Yifan Qiao@yifandotqiao·Nov 16

Thanks @digitalocean for the invite and nice recap! Hard to believe it’s only been a month, and we’re already seeing even crazier numbers every day. Let’s keep bringing @vllm_project to more chips!

0:47
vLLM avatar
⟳ Simon Mo retweeted
vLLM@vllm_project·Nov 16

.@NVIDIA put together a video walkthrough and deployment guide for running GLM-5.3 NVFP4 locally on two DGX Stations with vLLM. The setup keeps all model weights in GPU memory, using tensor parallelism across two GB300 GPUs to serve requests through a single OpenAI-compatible API. It also enables MTP speculative decoding, FP8 KV cache, and prefix caching. Watch the walkthrough: youtube.com/watch?v=opRXPRnS… Get the deployment guide: developer.nvidia.com/blog/wh…

Inferact avatar
⟳ Simon Mo retweeted
Inferact@inferact·Nov 16

With latency covered, what remains is throughput, which depends on how often the GPUs wait for each other. TP2 all-reduces twice in every layer, taking 18% of its GPU time. DEP2’s ranks meet in every MoE layer, so a 4:2 request split cost it 15%. PP2 hands off once per batch. 3/5

Post attachment
Inferact avatar
⟳ Simon Mo retweeted
Inferact@inferact·Nov 16

An agent turn adds a few thousand tokens to a long, mostly cached context, then spends seconds decoding and calling tools, so a few seconds of TTFT is enough. We tested a 100K-token cached prefix plus 7.5K–12.5K new tokens. PP2’s light-load TTFT was 0.45 s against TP2’s 0.31 s. 2/5

Post attachment
Inferact avatar
⟳ Simon Mo retweeted
Inferact@inferact·Nov 16

Pipeline parallelism (PP) is rarely picked for prefill because of its pipeline bubbles. We think agentic serving changes that. On @MiniMax_AI MiniMax-M3.1-Flash with @vllm_project on two NVIDIA B300s, PP2 sustained 23% more requests per GPU than DEP2 and 17% more than TP2 at a 2 s p50 TTFT target. 1/5

Post attachment
Asuka Zheng🦭 avatar
⟳ Simon Mo retweeted
Asuka Zheng🦭@VoidAsuka·Nov 16

“The last thing a model sees before being captured in a cuda graph.”

Post attachmentPost attachment