Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gui…
What if an agent could decide how to structure its own computation?
Agentic Meta-Reasoning: Let the model decide what to explore, what to build on, what to verify, and where to spend its next unit of compute: effectively constructing its own computational graph as it reasons. 🧵
Paras Dahal, @anton_bakhtin , @TacoCohen , Zhengxing Chen, Carole-Jean Wu, Rob Fergus, Scott Yih, @syhw , @rsalakhu , @prfsanjeevarora , @jaseweston
💥I have been able to independently verify @mc_mozer and Google team's work on ♾ Recirculation👀 using Llama 3.2 1B model on a M4 MacBook.
😱 +16.95% improvement on GSM8K-Platnium with zero weight changes.
Reproduction code with optimized mlx prefill kernel. 👇
We introduce an inference-time recurrence mechanism that allows off-the-shelf, frozen LLMs to act as dynamical systems. By leaking deep representations back to shallow layers, we see bonkers gains—23% ⬇️ in perplexity, 21% ⬆️ rel. accuracy in GSM8k— w/o touching model weights.
We introduce an inference-time recurrence mechanism that allows off-the-shelf, frozen LLMs to act as dynamical systems. By leaking deep representations back to shallow layers, we see bonkers gains—23% ⬇️ in perplexity, 21% ⬆️ rel. accuracy in GSM8k— w/o touching model weights.
What if a foundation model could tell us how to modify its architecture to boost inference and reasoning instantaneously—without retraining? What if that tweak incurred near zero latency cost during generation and supported indefinite state tracking? arxiv.org/abs/2608.17981
Interesting work! Their central result that RoPE struggles to distinguish positions and tokens in long contexts is very closely related to our PoPE paper, where we analysed RoPE entangles “what” and “where” in attention and proposed a decoupled alternative.
PoPE: arXiv 2509.10534
Excited to share our new paper: RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
LLMs often fail on inputs well within their advertised context lengths. We show that these failures are not merely engineering issues, but from intrinsic limitations of RoPE in long contexts.
Main finding: In long contexts, RoPE-based attention frequently assigns the same attention weight to a token even when it is moved to different positions. Similarly, it can assign the same attention weight to different tokens at the same position.
In this sense, RoPE attention fails to distinguish both where a token appears and what token appears there — hence the title.
We prove these results theoretically and verify them empirically. While the theoretical analysis focuses on a single attention head, we complement it with experiments on real multi-layer, multi-head LLMs.
The experiments confirm failures predicted by our theory: LLMs optimized for needle-in-a-haystack-style retrieval will inevitably struggle on a very simple task that asks for the k-th item in a list.
My personal takeaway: advertised context lengths should be interpreted with care. Future long-context LMs may require rethinking how position and token order are represented. With current architectures, agentic frameworks that break long contexts into shorter ones may be a more effective way to work around the intrinsic limitations of RoPE.
Paper: arxiv.org/abs/2605.15514
Huge congrats to my student Yufeng Du and others!
New preprint: The Self Requires Learning. Self-consciousness requires continual learning + world-modeling. I introduce "bounded integration" to connect perspective, identity, and self-representation — and diagnose what current AI systems have and lack.
[5/5] We introduce a taxonomy of recurrent models defined by their recurrence axis and their ratio of input tokens to recurrence steps. The next generation of efficient foundation models need to bridge the gap between a transformer's parallelism and the brain's dynamical nature.
[4/5] Recurrence is an obvious solution, but the literature often conflates qualitatively different varieties of recurrence in transformers, some of which enable indefinite state tracking and others (particularly, variants of looped transformers) do not.
[3/5] Because activations in the transformer flow in one direction, evolving state representations are pushed deeper into their layer stack. As a result, vital information is inaccessible to shallower layers, and ultimately the model’s depthis exhausted.
[1/5] Intelligent behavior relies on maintaining an evolving, dynamic model of the environment. But even frontier models sometimes fail at tracking state in dialogs, e.g., actual transcript from 4/20/2026:
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
[3/4] To train the model to calibrate its uncertainty and use <don't know> outputs judiciously, we frame the selection of each output token as a sequential-decision problem with a time penalty. We refer to the class of methods as “Catch Your Breath” losses.
[2/4] The model can request additional compute steps for any token by emitting a <don't know> output. If the model is granted a delay, a <pause> token is inserted at the next input step, providing the model with additional compute resources to generate an output.