←
Tri Dao1,015 posts
Tri Dao avatar✓

Tri Dao ✓

@tri_dao

Asst. Prof @PrincetonCS, Chief Scientist @togethercompute. Machine learning & systems.

◷ Joined May 2012▧ 59 Photos and videos
Tri Dao avatar
Tri Dao@tri_dao·11 Jul 2024

FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS! 1/

Post attachment
Alpin avatar
⟳ Tri Dao retweeted
Alpin@AlpinDale·Nov 7

Very exciting to finally show this off: lossless compression of BF16 models to 12bits (based on NyoomFloat12 from @togethercompute). A lot of work went into this to make it as efficient to run as possible, since no hardware has native support for this dtype. github.com/dphnAI/nf12

Post attachment
Together AI avatar
⟳ Tri Dao retweeted
Together AI@togethercompute·Nov 4

Inference is entering a new era. Models ship weekly, traffic shapes shift daily, and hardware turns over every year. The era of Static Inference, tuned once for one model with one traffic shape on one chip, is ending. Welcome to the era of Polymorphic Inference: inference that adapts to new models, workloads, and hardware as they arrive. Today we’re announcing Together Compiler, our first step into that era. It enables continual optimization, so a single overnight run can match or beat the manual expert tuning that used to take weeks. Read the full vision from our SVPs of Research, @simran_s_arora and @realDanFu, on Polymorphic Inference and our research roadmap 👇 togetherai.link/uQJRZjW

0:14
SemiAnalysis avatar
⟳ Tri Dao retweeted
SemiAnalysis@SemiAnalysis_·Nov 4

The Together Compiler (TKCC) from @simran_s_arora and her team is honestly one of the coolest things ever, and we have validated one of the InferenceX/AgentX results! The ability to continuously improve performance automatically while ensuring accuracy is great enough to quickly deploy to production! Simran and her team are using TKCC and working on applying it to even more use cases. They will share more soon! x.com/simran_s_arora/status/…

Dan Fu avatar
⟳ Tri Dao retweeted
Dan Fu@realDanFu·Nov 4

Super excited to (officially) share Together Compiler with the world! This is our opinionated take at @togethercompute about how to scale inference in an increasingly polymorphic world (many models, archs, traffic, hardware). Deployed across our inference fleet today 🚀 1/

Tri Dao avatar
Tri Dao@tri_dao·Nov 5

Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share

James Whittington avatar
⟳ Tri Dao retweeted
James Whittington@jcrwhittington·Nov 2

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context. Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster. With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington

0:12
Sylvie L avatar
⟳ Tri Dao retweeted
Sylvie L@__sylvie_l__·Oct 29

Lossless weight compression has saved LLMs GPU memory, but not time. Decoding costs more than it saves. NyoomFloat12 (NF12) is a lossless 12-bit BF16 format with decode cheap enough to speed up inference. Qwen3-32B: 1.23× faster token generation, up to 2.54× rollout throughput🧵

Post attachment
Tri Dao avatar
Tri Dao@tri_dao·Oct 28

Super interesting result: H-Net learns to dynamically tokenize variable names in coding data as separate tokens, and it scales much better on code compared to BPE

1:19
driss guessous avatar
⟳ Tri Dao retweeted
driss guessous@drisspg·Oct 23

Gave a talk at PTC on flex_gemm which is a new gemm_epilogue API in pytorch built on Quack. I think its neat, not ground breaking but neat nonetheless :) github.com/drisspg/transform… planning to bootstrap this and continue to add fun examples

Ali Hatamizadeh avatar
⟳ Tri Dao retweeted
Ali Hatamizadeh@ahatamiz1·Oct 17

I agree with one thing. Making linear attention work at scale takes talent, and not every lab has it. The rest doesn’t hold up. In order: (1) “Why are the Western labs choosing to drop context instead of compressing it?” The first rigorous, controlled, large-budget test of compression was what we did at NVIDIA’s in June 2024(arxiv.org/abs/2406.07887). We trained 8B Mamba, Mamba-2 and Transformer models on the same data for up to 3.5T tokens. Additionally we trained an 8B hybrid of 43% Mamba-2, 7% attention and 50% MLP and we showed it beats the 8B Transformer on all 12 standard tasks (+2.65 points on average). Interestingly this hybrid variant was up to 8× faster at generation, and matched or beat the Transformer across 23 additional long-context tasks. This is seven months before MiniMax-01, fifteen before Qwen3-Next, sixteen before Kimi Linear. And since then NVIDIA has been shipping a hybrid architecture, with modifications, repeatedly. Good examples are Nemotron-H, Nemotron Nano 2, and Nemotron 3 (Nano, Super, Ultra) which is a Mixture-of-Experts hybrid Mamba-Transformer family with context up to 1M tokens, post-trained with multi-environment RL. Nano keeps exactly 6 attention layers out of 52 and the other 46 are Mamba-2 and MoE blocks. Super is 120B total / 12B active, pre-trained on 25 trillion tokens. IBM’s Granite 4.0 (October 2025) interleaves Mamba-2 and attention layers at 9:1. To this list, you can also add Jamba (AI21, March 2024, a 52B production hybrid), RecurrentGemma (Google, April 2024), Codestral Mamba (Mistral, July 2024) and Phi-4-mini-flash (Microsoft, July 2025). (2) “SWA is a lossless local transformation, therefore superior.?” SWA is lossless inside the window and zero outside of it and it is exactly why it fails so badly. (arxiv.org/abs/2509.24552) for example shows that larger sliding windows hurts the long-context performance. This is because a precise local softmax lets the model avoid training the long-term memory it would otherwise rely on. They show that a hybrid with a 128-token window beat the same hybrid with a 2048 window by 16 NIAH points (88.9% higher recall), and at 131k the short-window models still retrieved about 30% of needles while the 2048-window model was near 0%. And the same degradation from larger training windows shows up in local-global stacks that alternate SWA with full attention ( e.g. the GPT-OSS/Gemma recipe). (3) “Compression is strictly harder to serve.” This was true in 2024. But now vLLM, FlashInfer, SGLang and TensorRT-LLM serve hybrid linear models extensively and without any issues. The most exciting part is the recent FlashInfer additions that let you use chunked gated-delta-rule prefill and gated-delta-rule decode kernels, with an SM100 (Blackwell) path. It even supports much newer models like Gated DeltaNet-2 and also lets you use efficient kernels for training. Also, let's not forget ReplaySSM (arxiv.org/abs/2610.13778) which enables a very optimized speculative decoding path for linear attention. (4) “If you write all the kernels anyway, why not roll your own attention in the meantime?” On NVIDIA hardware, linear attention is just as free as attention. The Mamba CUDA kernels exist. flash-linear-attention, the Triton library that the GDN and KDA layers in Qwen3-Next, Qwen3.5 and Kimi Linear use, was written for NVIDIA GPUs, and vLLM and SGLang use it directly instead of reimplementing it. Megatron has supported Mamba-2 training since 2024. A Western lab on H100s that wants a hybrid doesn’t need to write a kernel. It can just import it. So kernel cost can’t be what keeps them on using the SWA. Also consider that decode is memory-bandwidth-bound, so the cost of every generated token is dominated by reading the KV cache, which grows with context length. A constant-size state removes that growth. That’s why Nemotron is mostly Mamba-2, and why Nemotron 3 Super gets 2.2× the throughput of GPT-OSS-120B at 8K in / 16K out. (5) “For your first release, it’s a much safer option to ship SWA. Everyone knows it’s working just fine.”? Safe compared to what? “Safe” should mean proven in production at scale. By that standard the hybrids are the safe choice and SWA is the bet. Look at what has shipped on Gated DeltaNet alone. Qwen3-Next, then Qwen3.5 from 27B up to 397B-A17B, then Qwen3.6. Kimi Linear, then Kimi K3 which is a 2.8 trillion parameters, open weights, 1M-token context, a 3:1 hybrid of Kimi Delta Attention and gated MLA. GLM-5.3-Flash on the same KDA layout. KDA is Gated DeltaNet with channel-wise instead of scalar gating. Right now, three frontier labs run their model in production with 1M context, RL post-training, etc. all on the same layer. Meanwhile SWA is the architecture with the documented issue. It fails past the window and drags any hybrid it sits in down to pure-SWA recall. And when MiniMax reached for an SWA hybrid as the fallback in M2, it failed and they went to full attention instead. In my view, shipping SWA for a first release in late 2026 isn’t conservative. It’s picking the layer with a known failure mode because it needs no new code.

doubleAI avatar
⟳ Tri Dao retweeted
doubleAI@doubleAI·Oct 19

What if code could think while it runs? Meet Maxwell: the world’s first Code Execution Model. A new species of AI that lives inside the machine. We believe it’s the key to unlocking 100% of compute. Requests for early access are open, link below.

1:41
Tri Dao avatar
Tri Dao@tri_dao·Oct 19

We now have a platform for SFT & RL. This is why open weight models are great

1:35
Tri Dao avatar
Tri Dao@tri_dao·Oct 19

I’ve often wanted to pair a smart orchestrator / architect model with a super fast worker model. Diffusion LLM could fill the worker role

0:30
Together AI avatar
⟳ Tri Dao retweeted
Together AI@togethercompute·Oct 15

.@NVIDIA Nemotron 3.5 Super VL is now live on Together AI. An open-weight multimodal model built for visual reasoning and always-on agents that see, reason, and act across documents, screens, and video.

Post attachment