
Joint work with @qyndrmn, @YochaiBlau, @chmiel_brian, @RonBanner1, @soudry_daniel and Boris Ginsburg @nvidia

Joint work with @qyndrmn, @YochaiBlau, @chmiel_brian, @RonBanner1, @soudry_daniel and Boris Ginsburg @nvidia

💡 Takeaway: evidence-sparse long-context tasks and retrieval are one problem at different scales, and a single model can solve both. Bigger context windows aren't enough. Models should be trained to recall.

⚡ Massive speedup Quadratic prefill becomes linear scaling. Chunks are encoded independently, and generation only sees the selected text. → Fewer FLOPs beyond ~18K tokens → Lower time-to-first-token from ~32K tokens → The gap keeps widening as context grows


📏 Extreme scale One selector across six orders of magnitude, from 8K-token prompts to a 3B-token, 21M-chunk Wikipedia index. All 4 backbones beat SOTA retriever + reranker pipelines (recall@10): • HotpotQA: 49.1% → 73.2% • 2Wiki: 31.7% → 60.1% • MuSiQue: 8.8% → 14.4%

🧠 Unified architecture No external retriever. With <500K additional parameters (<0.005% of the model), we convert a frozen LLM into its own retriever. The same model ranks the evidence, then answers. Works on dense, linear-attention and Mamba hybrids.

RAG and long context are usually treated as rival approaches, but both pick the evidence that matters, just at different scales. What if one LLM did both? UNREAL: one frozen LLM retrieves from a 3B-token corpus and filters its own 128K prompt. arxiv.org/abs/2610.08463

Accepted to NeurIPS 2026!
Excited to share our new arXiv preprint: "Retrieval from Within: An Intrinsic Capability of Attention-Based Models" We introduce INTRA, a framework where attention-based models retrieve from their own internal representations. arxiv.org/abs/2605.05806 1/5 🧵

Thrilled to share our new paper: "LLM Explainability with Counterfactual Chains and Causal Graphs"! 🚀 We introduce a fully automated, model-driven method to extract global, concept-level causal graphs of an LLM's internal reasoning. 📄 arxiv.org/abs/2606.05972 🧵👇 [1/8]

Huge thanks to my wonderful co-authors at Nvidia: Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, and Boris Ginsburg. Looking forward to feedback, questions, and discussions! Paper: arxiv.org/abs/2605.05806 5/5 🧵

On QA benchmarks, INTRA improves both complete-evidence recall and end-to-end answer quality, especially on multi-hop settings where assembling the right evidence matters. It also amortizes context encoding by reusing precomputed encoder states. 4/5 🧵

The key idea is simple: pre-encode evidence chunks once, then let decoder attention score and select relevant chunks directly. This keeps retrieval and generation in the same representation space, reducing the retriever-generator mismatch in standard RAG. 3/5 🧵

RAG usually separates retrieval and generation: one model retrieves text, another re-encodes it and generates. INTRA asks a different question: what if the generator’s own decoder attention queries can retrieve the evidence it needs? 2/5 🧵

Excited to share our new arXiv preprint: "Retrieval from Within: An Intrinsic Capability of Attention-Based Models" We introduce INTRA, a framework where attention-based models retrieve from their own internal representations. arxiv.org/abs/2605.05806 1/5 🧵

Data Movement Is All You Need: A Case Study on Optimizing Transformers by Andrei Ivanov (ETH Zurich); Nikoli Dryden (ETH Zurich)*; Tal Ben-Nun (ETH Zurich); Shigang Li (ETH Zurich); Torsten Hoefler (ETH Zürich)

Our new work on training convnets using random image sizes: arxiv.org/abs/1908.08986