
𝗝𝗲𝗳𝗳 𝟭.𝟭 𝗶𝘀 𝗼𝘂𝘁! 🚀 Our open-source Jeff decision models just got a lot better. Jeff-1.1-Large on OpenDecisions: 89.0 in one pass 92.3 with optional reasoning (Jeff-1.0-Large: 84.9, Jev: 80.1) What changed? 🧵👇

Applied Math PhD @ Harvard & Kempner Institute

𝗝𝗲𝗳𝗳 𝟭.𝟭 𝗶𝘀 𝗼𝘂𝘁! 🚀 Our open-source Jeff decision models just got a lot better. Jeff-1.1-Large on OpenDecisions: 89.0 in one pass 92.3 with optional reasoning (Jeff-1.0-Large: 84.9, Jev: 80.1) What changed? 🧵👇

🚀 Our technical report for our K2 horizon models is out! It is the most detailed technical report of any open-source models that has ever been released. If you've ever wanted to see all the details that go into LLM training across all training phases, here it is!
Looking for a reference manual for building a frontier LLM? We wrote one 👇 The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today. • Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters. • Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up. • Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking. • Post-training: we document our full production-scale run for the 375B flagship. • Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2× faster decoding, lossless by construction, no draft model. This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them. More details: arxiv.org/abs/2611.21003

⚡NEW: Unlocking lossless speedups in LLMs via discrete diffusion. A new diffusion-augmented model, Uno, combines autoregressive and diffusion weights in one network to generate multiple tokens per forward pass. More in today’s Deeper Learning blog post: bit.ly/4AmooR0

Great blog post by @JonathanGeuter and @KempnerInst on our recent Uno paper. Autoregressive LLMs + Diffusion adapters >>> Training Diffusion LLMs from scratch or Speculative Decoding with small drafters Stay tuned for a follow-up to the Uno paper with comparisons against DFlash2 and DSpark. kempnerinstitute.harvard.edu…
Diffusion LLMs have two limitations relative to AR models: (1) Lower quality, and (2) Slower inference at large batch sizes. We address this "Uno" > Retains the AR architecture of LLMs > Each layer has two sets of weights: AR weights and Diffusion weights > Diffusion weights enable parallel sampling from the AR distribution losslessly Results: 💥Faster than all speculative decoding methods: DFlash and EAGLE-3 🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada Links to the paper, models, code below

Great work by Daniel, happy to see people use our decision model training dataset! Check it out here: huggingface.co/datasets/jgeu…
Some things that were useful when my agent and I were building the data for this model: - Check how well you can guess the answer from the options alone. One source looked like a +4.5 gain, but it was mostly "the longest option is right". Real gain was more like +2.3 - Weight wrong options by how common they are. arXiv labels with uniform wrong options got +18 over chance without reading any text - Keep one writing style across classes. With one class written by humans and the other by an LLM, the model just learned to spot LLM style - Shuffle the options per example. Models that read all the options together can flip answers when you reorder them (12-23% in Crivo) - Try different mixes, but with more than one seed. Single runs differed by up to ~8 points on one test set. Adding diverse sources gave the same lift as 4x the rows and steps, at about a quarter of the compute - Check what your trainer silently drops. Ours skipped 53k one-label rows, the only verification data in the mix Data, with source, labeller and licence on every row: huggingface.co/datasets/dava…

@vanstriendaniel might be of interest to you! We've also released our training data now.

📄 Paper: arxiv.org/abs/2610.17205 (will be updated to v1.1 soon!) 🤗 Models: huggingface.co/collections/j… 📊 Data: huggingface.co/datasets/jgeu… ⚙️ Code: github.com/j-geuter/Jeff Joint work with Andrew Cheng and @elmelis.

𝗢𝗻 𝗝𝗲𝘃𝗕𝗲𝗻𝗰𝗵, 𝗼𝘂𝗿 𝟭𝟮𝗕 𝘀𝗰𝗼𝗿𝗲𝘀 𝟴𝟭.𝟵, which makes it the 𝗼𝗻𝗹𝘆 𝗼𝗽𝗲𝗻 𝗺𝗼𝗱𝗲𝗹 𝗶𝗻 𝘁𝗵𝗲 𝘁𝗼𝗽 𝟭𝟯 𝘀𝗺𝗮𝗹𝗹𝗲𝗿 𝘁𝗵𝗮𝗻 𝟯𝟭𝗕 and puts it 𝟰.𝟴 𝗽𝗼𝗶𝗻𝘁𝘀 𝗮𝗯𝗼𝘃𝗲 𝗝𝗲𝘃: benchmarkheaven.com/jev-mode…. Scores on our other models will follow!

Every model size improves, the small ones the most (OpenDecisions Capability, without reasoning): 27B: 81.4 → 86.8 12B: 74.5 → 85.0 4B: 66.7 → 79.3

5️⃣ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 Without reasoning, mean latency per decision is between 25ms (on the 12B) and 63ms (on the 27B), compared against ~200ms through the Jev API.

4️⃣ 𝗢𝗽𝗲𝗻 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 We release Jeff 1.1's training data: 261,561 decision questions across various domains and question types. To our knowledge, this is the first open-source release of decision model training data at this scale. Train your own decision model!

3️⃣ 𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 When unsure, Jeff can now reason: the base model (without LoRA) writes a short reasoning, and Jeff (with LoRA) reads it in a second forward pass. Confident questions still take only one pass.

2️⃣ 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝘁𝗿𝗶𝗰𝗸𝘀 - 1,500 instead of 400 training steps - model soup over 3 training runs - bidirectional attention on the 4B, 12B, and 27B models (surprisingly, bidirectional attention did not help the 31B)

1️⃣ 𝗕𝗲𝘁𝘁𝗲𝗿 𝗱𝗮𝘁𝗮 We added about 86,000 new training questions written and labeled by strong teacher LLMs, or computed programmatically. We also relabeled the 1.0 training data with a strong teacher.

Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch. No diffusion teacher. No flow map. No CFG. No JVP. We believe this work will challenge how we think about building one-step models. Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han. 🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L 🎨 Scales to text-to-image generation (GenEval 0.82) 🤖 World modeling on RT-1 / Bridge-V2 Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].

📄 𝗣𝗮𝗽𝗲𝗿: arxiv.org/abs/2610.17205 🤗 𝗠𝗼𝗱𝗲𝗹𝘀: huggingface.co/collections/j… 📊 𝗗𝗮𝘁𝗮𝘀𝗲𝘁: huggingface.co/datasets/jgeu… ⚙️ 𝗖𝗼𝗱𝗲: github.com/j-geuter/jeff-ser… Joint work with Andrew Cheng and @elmelis