it's hilarious how many have taken the bitter lesson to mean just scaling up transformers. it's about scaling, but innovations in algorithmic primitives can lead to much more efficient scaling
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gui…
💡 Who learns first can shape what the whole model learns.
If you train hybrid LLMs, we’d love to hear whether you observe similar competition!
📄 Paper: arxiv.org/abs/2610.26830
💻 Code: github.com/thunlp/competing-…
(5/5)
🛠️ Better Layer Division (BLD):
❄️ Freeze recurrent blocks for the first ~9% of pretraining tokens.
🔓 Then unfreeze and train jointly.
This gives attention a head start in retrieval.
Same architecture, data and token budget.
(3/5)
🧩 We call it Competing Learning.
When recurrence learns retrieval first, it reduces the prediction error—and the learning signal left for softmax attention.
🧪 Freeze recurrent KDA layers, and attention’s retrieval learning increases.
(2/5)
Stronger linear attention can make hybrid LLMs weaker. Why?
This weird finding has been on my mind for a long time. We finally have a hypothesis that could explain it, and experiments that support it.
Here’s the story. 🧵
(1/5)
a life update: i started a new company called Pragmatik (p7k) Labs (语用科技) in shanghai, focusing on the research of next-generation agents across digital and physical worlds. thanks to Gaorong Ventures and HSG (红杉中国 & 高榕创投) for co-leading this round, and to Tencent (腾讯) and Shanghai Engine Fund (上海未来产业基金) for the support. @pragmatik_labs · pragmatik.com
(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few?
Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token.
Why? RoPE has an inductive bias that works against exact copying.
Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.
I found out the other day that any compression tool can be contorted to do language modeling. Turns out gzip can generate text that somewhat *resembles* Shakespeare. Short write up linked below
The TurboQuant paper (ICLR 2026) contains serious issues in how it describes RaBitQ, including incorrect technical claims and misleading theory/experiment comparisons.
We flagged these issues to the authors before submission. They acknowledged them, but chose not to fix them. The paper was later accepted and widely promoted by Google, reaching tens of millions of views.
We’re speaking up now because once a misleading narrative spreads, it becomes much harder to correct. We’ve written a public comment on openreview (openreview.net/forum?id=tO3A…).
We would greatly appreciate your attention and help in sharing it.
Introducing TurboQuant: Our new compression algorithm that reduces LLM key-value cache memory by at least 6x and delivers up to 8x speedup, all with zero accuracy loss, redefining AI efficiency. Read the blog to learn how it achieves these results: goo.gle/4bsq2qI
We are disappointed to see that the TurboQuant team has not directly responded to our concerns majorly on Openreview. Their reply even suggests that we had not raised these technical points to them through academic channels over the past year, which is factually incorrect.
We have submitted our email records with the TurboQuant team to ICLR Chairs. According to ICLR Code of Ethics “Researchers must not deliberately make false or misleading claims, fabricate or falsify data, or misrepresent results. Methods and results should be presented in a way that is transparent and reproducible. ”, we respectfully request that ICLR initiates a formal research-integrity review of this paper.
The TurboQuant paper (ICLR 2026) contains serious issues in how it describes RaBitQ, including incorrect technical claims and misleading theory/experiment comparisons.
We flagged these issues to the authors before submission. They acknowledged them, but chose not to fix them. The paper was later accepted and widely promoted by Google, reaching tens of millions of views.
We’re speaking up now because once a misleading narrative spreads, it becomes much harder to correct. We’ve written a public comment on openreview (openreview.net/forum?id=tO3A…).
We would greatly appreciate your attention and help in sharing it.
An open-source 9B model with a 1M-token context and an Apache 2.0 license has just been released on Hugging Face. It’s designed to run on a single consumer-class GPU, such as the NVIDIA RTX 5090.
This model breaks the "Compute Wall" and the "Memory Wall," achieving 3.5× faster inference and significantly lower KV-cache overhead compared to dense baselines.
This is no longer an "either-or" choice between performance and efficiency.
How?
Full Attention mechanism's computational complexity grows quadratically with length, making edge-side long-text inference "slow and memory-intensive."
Solution: MiniCPM-SALA adopts a golden ratio of 75% Linear Attention + 25% Sparse Attention.
MiniCPM-SALA (9B) is OpenBMB’s long-context model aimed at running 1M to 2M tokens on a single GPU without the memory spikes and OOM failures common with dense full attention. The main idea is a sparse plus linear hybrid that keeps long-range recall accurate while keeping cost manageable as context grows.
- Architecturally, about 25% of layers use InfLLM-V2 style sparse attention for high-fidelity long-range retrieval, while about 75% use Lightning linear attention, so compute scales close to linearly with sequence length. Instead of a uniform interleave, the sparse layers are placed via a 1:3 layer-selection pattern.
- For positional handling and stability, SALA uses hybrid positional encoding (HyPE): RoPE stays in the linear layers but is removed in sparse layers to avoid long-range decay, and it adds QK-normalization plus output gating to improve stability and reduce attention-sink behavior.
- Training is done by converting a pretrained Transformer, not training from scratch. It starts from a MiniCPM-4.0 intermediate checkpoint trained on 7T tokens, then applies HALO conversion, keeping the 1st and last layers unconverted and initially training only the converted linear layers.
Conversion plus post-training totals about 2T tokens, framed as about a 75% cost reduction versus an 8T scratch run, with context ramping from 512 to 4K, then to 32K, 160K, and 520K, followed by SFT at 64K and 140K.
Reported results keep standard performance strong (76.53 average, HumanEval 95.12, AIME24 83.75, AIME25 78.33)
While improving long-context behavior (RULER 92.65 at 64K, 89.37 at 128K). It also reports single-GPU 1M-token inference where Qwen3-8B OOMs, 256K TTFT, improving from 180.8s to 51.6s, and RULER holding at 86.3 at 1M and 81.6 at 2M without YaRN.
Go to Hugging Face/GitHub to test the model capabilities yourself.