Yesterday, we shared the details on our xLSTM 7B architecture. Now, let's go one level deeper🧑🔧
We introduce
⚡️Tiled Flash Linear Attention (TFLA), ⚡️
A new kernel algorithm for the mLSTM and other Linear Attention variants with Gating.
We find TFLA is really fast!
🧵(1/11)
Introducing e-means, an algorithm beating k-means in 10x fewer epochs.
With order of magnitude wall-clock speedups, e-means is released in FAISS and needs no hyperparameter tuning beyond k. It is robust to various scales and datasets. Still skeptical ? Just try e-means [1/10]🧵
Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.
Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.
With @kubicek_ales@caciolai@PengLiangzu@SukritSumant@Ben_Hoov@FraserGreenlee@RMBattleday@radim_urban@thoefler@jcrwhittington
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
We’re sharing Dist-MoE, open-source library from Meta SuperIntelligence Labs (MSL @AIatMeta ). Dist-MoE provides fused communication and compute kernels for BF16/MXFP8 training and inference, plus NVFP4 inference.
✅ Fused communication and compute kernels
✅ Full CUDA graph support
✅ Dynamic activation packing across layers into a bounded buffer
✅ Dynamic, capacity-driven recompute
✅ Forward and backward kernels for BF16/MXFP8
✅ No shared-expert branch required
Some highlights from our GB300 NVL72 benchmarks:
• Up to 1.40× speedup over Mixture-of-Kittens (MoK) in our imbalanced, ten-layer MXFP8 training workload across 64 GPUs. Dist-MoE packed activation state across all ten layers with zero capacity-induced recompute.
• 3.09 useful PFLOP/s per GPU (3,090 TFLOP/s) for NVFP4 Mega inference across 16 GPUs.
• 1.86 useful PFLOP/s per GPU (1,856 TFLOP/s) for MXFP8 Staged training across 64 GPUs.
• 1.27 useful PFLOP/s per GPU (1,271 TFLOP/s) for BF16 training across 64 GPUs.
• Sub-linear scaling under imbalance: in NVFP4 inference across 16 GPUs, 8× the received rows on the busiest GPU required 4.82× the iteration time; about 40% lower iteration time per received row.
Full configurations, raw results, and measurement definitions are in the blog.
@levendlee and I also put together an interactive visual explainer covering token dispatch and combine, quantization, forward/backward kernels, activation packing, dynamic recompute, and memory planning.
Blog:
meta-pytorch.org/dist_moe/pr…
Repository:
github.com/meta-pytorch/dist…
TorchTitan (@PyTorch ) integration:
github.com/pytorch/torchtita…
If you work on MoE training, inference, or GPU kernels, take a look. We welcome feedback and contributions.
#PyTorch#MixtureOfExperts#MoE#OpenSource#TorchTitan#DistMoE
I agree with one thing. Making linear attention work at scale takes talent, and not every lab has it. The rest doesn’t hold up. In order:
(1) “Why are the Western labs choosing to drop context instead of compressing it?”
The first rigorous, controlled, large-budget test of compression was what we did at NVIDIA’s in June 2024(arxiv.org/abs/2406.07887). We trained 8B Mamba, Mamba-2 and Transformer models on the same data for up to 3.5T tokens. Additionally we trained an 8B hybrid of 43% Mamba-2, 7% attention and 50% MLP and we showed it beats the 8B Transformer on all 12 standard tasks (+2.65 points on average). Interestingly this hybrid variant was up to 8× faster at generation, and matched or beat the Transformer across 23 additional long-context tasks.
This is seven months before MiniMax-01, fifteen before Qwen3-Next, sixteen before Kimi Linear. And since then NVIDIA has been shipping a hybrid architecture, with modifications, repeatedly. Good examples are Nemotron-H, Nemotron Nano 2, and Nemotron 3 (Nano, Super, Ultra) which is a Mixture-of-Experts hybrid Mamba-Transformer family with context up to 1M tokens, post-trained with multi-environment RL. Nano keeps exactly 6 attention layers out of 52 and the other 46 are Mamba-2 and MoE blocks. Super is 120B total / 12B active, pre-trained on 25 trillion tokens. IBM’s Granite 4.0 (October 2025) interleaves Mamba-2 and attention layers at 9:1. To this list, you can also add Jamba (AI21, March 2024, a 52B production hybrid), RecurrentGemma (Google, April 2024), Codestral Mamba (Mistral, July 2024) and Phi-4-mini-flash (Microsoft, July 2025).
(2) “SWA is a lossless local transformation, therefore superior.?”
SWA is lossless inside the window and zero outside of it and it is exactly why it fails so badly. (arxiv.org/abs/2509.24552) for example shows that larger sliding windows hurts the long-context performance. This is because a precise local softmax lets the model avoid training the long-term memory it would otherwise rely on. They show that a hybrid with a 128-token window beat the same hybrid with a 2048 window by 16 NIAH points (88.9% higher recall), and at 131k the short-window models still retrieved about 30% of needles while the 2048-window model was near 0%. And the same degradation from larger training windows shows up in local-global stacks that alternate SWA with full attention ( e.g. the GPT-OSS/Gemma recipe).
(3) “Compression is strictly harder to serve.”
This was true in 2024. But now vLLM, FlashInfer, SGLang and TensorRT-LLM serve hybrid linear models extensively and without any issues. The most exciting part is the recent FlashInfer additions that let you use chunked gated-delta-rule prefill and gated-delta-rule decode kernels, with an SM100 (Blackwell) path. It even supports much newer models like Gated DeltaNet-2 and also lets you use efficient kernels for training. Also, let's not forget ReplaySSM (arxiv.org/abs/2610.13778) which enables a very optimized speculative decoding path for linear attention.
(4) “If you write all the kernels anyway, why not roll your own attention in the meantime?”
On NVIDIA hardware, linear attention is just as free as attention. The Mamba CUDA kernels exist. flash-linear-attention, the Triton library that the GDN and KDA layers in Qwen3-Next, Qwen3.5 and Kimi Linear use, was written for NVIDIA GPUs, and vLLM and SGLang use it directly instead of reimplementing it. Megatron has supported Mamba-2 training since 2024. A Western lab on H100s that wants a hybrid doesn’t need to write a kernel. It can just import it. So kernel cost can’t be what keeps them on using the SWA.
Also consider that decode is memory-bandwidth-bound, so the cost of every generated token is dominated by reading the KV cache, which grows with context length. A constant-size state removes that growth. That’s why Nemotron is mostly Mamba-2, and why Nemotron 3 Super gets 2.2× the throughput of GPT-OSS-120B at 8K in / 16K out.
(5) “For your first release, it’s a much safer option to ship SWA. Everyone knows it’s working just fine.”?
Safe compared to what? “Safe” should mean proven in production at scale. By that standard the hybrids are the safe choice and SWA is the bet.
Look at what has shipped on Gated DeltaNet alone. Qwen3-Next, then Qwen3.5 from 27B up to 397B-A17B, then Qwen3.6. Kimi Linear, then Kimi K3 which is a 2.8 trillion parameters, open weights, 1M-token context, a 3:1 hybrid of Kimi Delta Attention and gated MLA. GLM-5.3-Flash on the same KDA layout.
KDA is Gated DeltaNet with channel-wise instead of scalar gating. Right now, three frontier labs run their model in production with 1M context, RL post-training, etc. all on the same layer. Meanwhile SWA is the architecture with the documented issue. It fails past the window and drags any hybrid it sits in down to pure-SWA recall. And when MiniMax reached for an SWA hybrid as the fallback in M2, it failed and they went to full attention instead.
In my view, shipping SWA for a first release in late 2026 isn’t conservative. It’s picking the layer with a known failure mode because it needs no new code.
Given Meta also shipped SWA, I decided to go deeper into the "is linear attention a Chinese psyop" rabbit hole. Plus, as a person who spent some time working on architectures.
1. I would frame the question the following way: “Why are the Western labs choosing to drop context instead of compressing it?" If you think about it this way, there is a clear advantage of SWA: it preserves the full signal from the local context, while linear attention compresses everything, losing some information about local context as well. Given that the local memory is roughly equal, as well as the time complexity and assuming that you believe in "only global signal layers and only local signal layers", SWA seems to be superior. It's a lossless "local signal" transformation. Thus, the discourse about the West not adopting a "superior" architecture is not entirely correct IMO.
2. Compression is strictly harder to serve. We saw this a lot internally; inference engineers hate it. One of the reasons is all the stuff is optimised around KV cache, and recurrent state is breaking lots of nice properties: prefix state now needs snapshots at block boundaries, spec dec needs per-token rollback. RL becomes problematic.
So why do the Chinese seem to adopt compression anyway?
3. Hardware but in a bit of a weird way. On Hopper/Blackwell, you already have FA; on Ascend, etc., one needs to write all the kernels in Triton or TileLang. If you write all the kernels anyway, why not roll your own attention in the meantime? You're hiring a lot of kernel people that need to write all this stuff.
4. Selection bias. If you take a closer look at the Wester labs that ship SWA (usually OSS ones, we don't know what Ant/OAI/GDM ship to the closed models), they are publishing their first or second model iterations. For your first release, it's a much safer option to ship SWA. Everyone knows it's working just fine. (This was our own motivation at Mellum). They may switch to something smarter down the line and I believe this already somewhat happened, take Nemotron for example. The only case I'm aware of people actually going backwards is Poolside??? I'd expect OSS labs to start digging into compression along with subsequent releases.
5. It seems that talent plays a big role. Linear attention field is historically pretty much Chinese and Chinese labs seem to borrow the ideas from each other. Wester labs are mostly GDM/OAI alums, which are likely used to SWA (see GPT-OSS and Gemma).
To summarise, I expect linear attention (and whatever DeepSeek is cooking) to start slowly being adopted in the West with the next model releases and when the inference infra catches up.
Today we’re releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
→ 816B total / 42B active parameters
→ 1M token context window
→ Natively multimodal across text, image, video, audio
🚨Our new paper is out: Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs
We present model casting, a mid-training recipe that drastically sparsifies the activations within the FeedForward Network layer of a transformer. Model casting consists of switching to an activation that outputs zeros for all negative inputs (R-S+) and using an L1 loss to set the required sparsity as a target at training time.
Delta-rule linear attention appears in LLMs as Gated DeltaNet (GDN; Qwen3.8) and Kimi Delta Attention (KDA; Kimi K3). KDA swaps GDN’s scalar gate for a channel-wise one, but does that improve recurrence expressivity? With 2 tweaks, KDA can do 2D rotations. GDN can’t. How? 🧵👇
1/ big announcement today: we will be releasing an open weight version of muse spark 1.2 soon.
we also are releasing muse glimmer, a 30B agentic model with open weights under apache 2.0. muse glimmer can run on 24GB of VRAM without losing agentic reliability. 🧵
Now suddenly everyone in AI is talking about loops, whereas control system engineers (aka the cybernetics program) had known for a long time that all intelligent systems, natural or artificial, learn and improve via closed-loop feedback.
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: discoveryloop.com
New paper "On Subquadratic Architectures: From Applications to Principles" 🙌
On commonsense & reasoning benchmarks, xLSTM, Mamba-2 & Gated DeltaNet perform nearly indistinguishably.
Therefore, we analyse them where structure genuinely matters: on code & time series.👇
We never really knew how to train nonlinear RNNs well… BPTT struggled with vanishing grads (no long-range memory) and sequential rollout (hard to parallelizable).
What if instead an oracle told us the optimal memory state m_t at each step? Then the RNN could do one-step supervised learning on (m_t, x_{t+1}) → m_{t+1} labels.
We call this Supervised Memory Training (SMT): a replacement for BPTT that trains RNNs without unrolling them. SMT is time-parallelizable and solves vanishing gradients.
Website: akarshkumar.com/smt/
arXiv: arxiv.org/abs/2606.06479