←
Elad Hoffer32 posts
E

Elad Hoffer

@eladhoffer
◷ Joined July 2014▧ 7 Photos and videos
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

💡 Takeaway: evidence-sparse long-context tasks and retrieval are one problem at different scales, and a single model can solve both. Bigger context windows aren't enough. Models should be trained to recall.

Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

⚡ Massive speedup Quadratic prefill becomes linear scaling. Chunks are encoded independently, and generation only sees the selected text. → Fewer FLOPs beyond ~18K tokens → Lower time-to-first-token from ~32K tokens → The gap keeps widening as context grows

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

🎯 The same module works inside the prompt, with no long-context training It ranks the chunks in the prompt, drops the distractors, and passes only the evidence to the model. • NoLiMa @128K: 1.0% → 24.8% • LV-Eval @256K F1: 49.97% → 54.66% • Best on HELMET-RAG

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

📏 Extreme scale One selector across six orders of magnitude, from 8K-token prompts to a 3B-token, 21M-chunk Wikipedia index. All 4 backbones beat SOTA retriever + reranker pipelines (recall@10): • HotpotQA: 49.1% → 73.2% • 2Wiki: 31.7% → 60.1% • MuSiQue: 8.8% → 14.4%

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

🧠 Unified architecture No external retriever. With <500K additional parameters (<0.005% of the model), we convert a frozen LLM into its own retriever. The same model ranks the evidence, then answers. Works on dense, linear-attention and Mamba hybrids.

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Oct 8

RAG and long context are usually treated as rival approaches, but both pick the evidence that matters, just at different scales. What if one LLM did both? UNREAL: one frozen LLM retrieves from a 3B-token corpus and filters its own 128K prompt. arxiv.org/abs/2610.08463

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·Sep 25

Accepted to NeurIPS 2026!

Excited to share our new arXiv preprint: "Retrieval from Within: An Intrinsic Capability of Attention-Based Models" We introduce INTRA, a framework where attention-based models retrieve from their own internal representations. arxiv.org/abs/2605.05806 1/5 🧵

Nirit Nussbaum avatar
⟳ Elad Hoffer retweeted
Nirit Nussbaum@NiritNussbaum·Jun 7

Thrilled to share our new paper: "LLM Explainability with Counterfactual Chains and Causal Graphs"! 🚀 We introduce a fully automated, model-driven method to extract global, concept-level causal graphs of an LLM's internal reasoning. 📄 arxiv.org/abs/2606.05972 🧵👇 [1/8]

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·May 8

Huge thanks to my wonderful co-authors at Nvidia: Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, and Boris Ginsburg. Looking forward to feedback, questions, and discussions! Paper: arxiv.org/abs/2605.05806 5/5 🧵

Elad Hoffer avatar
Elad Hoffer@eladhoffer·May 8

On QA benchmarks, INTRA improves both complete-evidence recall and end-to-end answer quality, especially on multi-hop settings where assembling the right evidence matters. It also amortizes context encoding by reusing precomputed encoder states. 4/5 🧵

Elad Hoffer avatar
Elad Hoffer@eladhoffer·May 8

The key idea is simple: pre-encode evidence chunks once, then let decoder attention score and select relevant chunks directly. This keeps retrieval and generation in the same representation space, reducing the retriever-generator mismatch in standard RAG. 3/5 🧵

Elad Hoffer avatar
Elad Hoffer@eladhoffer·May 8

RAG usually separates retrieval and generation: one model retrieves text, another re-encodes it and generates. INTRA asks a different question: what if the generator’s own decoder attention queries can retrieve the evidence it needs? 2/5 🧵

Post attachment
Elad Hoffer avatar
Elad Hoffer@eladhoffer·May 8

Excited to share our new arXiv preprint: "Retrieval from Within: An Intrinsic Capability of Attention-Based Models" We introduce INTRA, a framework where attention-based models retrieve from their own internal representations. arxiv.org/abs/2605.05806 1/5 🧵

Alex Dimakis avatar
⟳ Elad Hoffer retweeted
Alex Dimakis@AlexGDimakis·4 Apr 2021

Data Movement Is All You Need: A Case Study on Optimizing Transformers by Andrei Ivanov (ETH Zurich); Nikoli Dryden (ETH Zurich)*; Tal Ben-Nun (ETH Zurich); Shigang Li (ETH Zurich); Torsten Hoefler (ETH Zürich)