←
Jonathan Geuter60 posts
Jonathan Geuter avatar

Jonathan Geuter

@JonathanGeuter

Applied Math PhD @ Harvard & Kempner Institute

◷ Joined March 2024▧ 3 Photos and videos
Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

𝗝𝗲𝗳𝗳 𝟭.𝟭 𝗶𝘀 𝗼𝘂𝘁! 🚀 Our open-source Jeff decision models just got a lot better. Jeff-1.1-Large on OpenDecisions: 89.0 in one pass 92.3 with optional reasoning (Jeff-1.0-Large: 84.9, Jev: 80.1) What changed? 🧵👇

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·17h

🚀 Our technical report for our K2 horizon models is out! It is the most detailed technical report of any open-source models that has ever been released. If you've ever wanted to see all the details that go into LLM training across all training phases, here it is!

Looking for a reference manual for building a frontier LLM? We wrote one 👇 The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today. • Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters. • Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up. • Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking. • Post-training: we document our full production-scale run for the 375B flagship. • Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2× faster decoding, lossless by construction, no draft model. This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them. More details: arxiv.org/abs/2611.21003

0:44
Kempner Institute at Harvard University avatar
⟳ Jonathan Geuter retweeted
Kempner Institute at Harvard University@KempnerInst·Nov 9

⚡NEW: Unlocking lossless speedups in LLMs via discrete diffusion. A new diffusion-augmented model, Uno, combines autoregressive and diffusion weights in one network to generate multiple tokens per forward pass. More in today’s Deeper Learning blog post: bit.ly/4AmooR0

Subham Sahoo avatar
⟳ Jonathan Geuter retweeted
Subham Sahoo@ssahoo_·Nov 10

Great blog post by @JonathanGeuter and @KempnerInst on our recent Uno paper. Autoregressive LLMs + Diffusion adapters >>> Training Diffusion LLMs from scratch or Speculative Decoding with small drafters Stay tuned for a follow-up to the Uno paper with comparisons against DFlash2 and DSpark. kempnerinstitute.harvard.edu…

Post attachment

Diffusion LLMs have two limitations relative to AR models: (1) Lower quality, and (2) Slower inference at large batch sizes. We address this "Uno" > Retains the AR architecture of LLMs > Each layer has two sets of weights: AR weights and Diffusion weights > Diffusion weights enable parallel sampling from the AR distribution losslessly Results: 💥Faster than all speculative decoding methods: DFlash and EAGLE-3 🔥 Beats ALL diffusion LLMs: Mercury 2, Diffusion Gemma, Llada Links to the paper, models, code below

Post attachment
Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Nov 2

Great work by Daniel, happy to see people use our decision model training dataset! Check it out here: huggingface.co/datasets/jgeu…

Some things that were useful when my agent and I were building the data for this model: - Check how well you can guess the answer from the options alone. One source looked like a +4.5 gain, but it was mostly "the longest option is right". Real gain was more like +2.3 - Weight wrong options by how common they are. arXiv labels with uniform wrong options got +18 over chance without reading any text - Keep one writing style across classes. With one class written by humans and the other by an LLM, the model just learned to spot LLM style - Shuffle the options per example. Models that read all the options together can flip answers when you reorder them (12-23% in Crivo) - Try different mixes, but with more than one seed. Single runs differed by up to ~8 points on one test set. Adding diverse sources gave the same lift as 4x the rows and steps, at about a quarter of the compute - Check what your trainer silently drops. Ours skipped 53k one-label rows, the only verification data in the mix Data, with source, labeller and licence on every row: huggingface.co/datasets/dava…

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

𝗢𝗻 𝗝𝗲𝘃𝗕𝗲𝗻𝗰𝗵, 𝗼𝘂𝗿 𝟭𝟮𝗕 𝘀𝗰𝗼𝗿𝗲𝘀 𝟴𝟭.𝟵, which makes it the 𝗼𝗻𝗹𝘆 𝗼𝗽𝗲𝗻 𝗺𝗼𝗱𝗲𝗹 𝗶𝗻 𝘁𝗵𝗲 𝘁𝗼𝗽 𝟭𝟯 𝘀𝗺𝗮𝗹𝗹𝗲𝗿 𝘁𝗵𝗮𝗻 𝟯𝟭𝗕 and puts it 𝟰.𝟴 𝗽𝗼𝗶𝗻𝘁𝘀 𝗮𝗯𝗼𝘃𝗲 𝗝𝗲𝘃: benchmarkheaven.com/jev-mode…. Scores on our other models will follow!

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

Every model size improves, the small ones the most (OpenDecisions Capability, without reasoning): 27B: 81.4 → 86.8 12B: 74.5 → 85.0 4B: 66.7 → 79.3

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

5️⃣ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 Without reasoning, mean latency per decision is between 25ms (on the 12B) and 63ms (on the 27B), compared against ~200ms through the Jev API.

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

4️⃣ 𝗢𝗽𝗲𝗻 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 We release Jeff 1.1's training data: 261,561 decision questions across various domains and question types. To our knowledge, this is the first open-source release of decision model training data at this scale. Train your own decision model!

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

3️⃣ 𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 When unsure, Jeff can now reason: the base model (without LoRA) writes a short reasoning, and Jeff (with LoRA) reads it in a second forward pass. Confident questions still take only one pass.

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

2️⃣ 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝘁𝗿𝗶𝗰𝗸𝘀 - 1,500 instead of 400 training steps - model soup over 3 training runs - bidirectional attention on the 4B, 12B, and 27B models (surprisingly, bidirectional attention did not help the 31B)

Jonathan Geuter avatar
Jonathan Geuter@JonathanGeuter·Oct 26

1️⃣ 𝗕𝗲𝘁𝘁𝗲𝗿 𝗱𝗮𝘁𝗮 We added about 86,000 new training questions written and labeled by strong teacher LLMs, or computed programmatically. We also relabeled the 1.0 training data with a strong teacher.

Jaeyeon (Jay) Kim avatar
⟳ Jonathan Geuter retweeted
Jaeyeon (Jay) Kim@Jaeyeon_Kim_0·Oct 15

Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch. No diffusion teacher. No flow map. No CFG. No JVP. We believe this work will challenge how we think about building one-step models. Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han. 🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L 🎨 Scales to text-to-image generation (GenEval 0.82) 🤖 World modeling on RT-1 / Bridge-V2 Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].