Continuous Diffusion LMs went from niche to contender this year, but the recipe is shaky: why self-conditioning? How to embed tokens? 🤨
Our new work takes your one-hot flow from 4% to 50% on TinyGSM without self-conditioning, and explores the design space of CDLMs. 🔥🚀 (1/6)
⚠️ Numerical issues can go on silently for decades like in the softmax/CE! Similar issues are present in other widely-used divergences like the KL and JSD
I just posted this paper overviewing what they are and how to fix them:
📚 arxiv.org/abs/2611.18511
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Big reasoning speedups are here. 1,000× faster training and 280× less compute than any looped model. All done using a fully trainable energy from the 80s
Keep reading to hear how energy is the right paradigm for reasoning
With @HughBlayney@jcrwhittington@prefrontal_ai
🧵/10
Presenting “Monte Carlo Methods for Diffusion Model Sampling” at the Mostly Monte Carlo seminar tomorrow (Fri, 2:30pm) at PariSanté Campus
Alongside @__SimonCoste__ who will present “Importance Sampling”
mostly-monte-carlo.github.io…
I’ll be giving a tutorial on diffusion language models at AACL-IJCNLP 2026 on Nov. 10
w/ @Bollegala
🔗 dllm-aacl.github.io/
(Slides coming soon!)
#aacl2026@aaclmeeting
💭Grateful for the opportunity to organize my perspective on this fast-moving field and share it with the NLP community. This will be my first tutorial, so it’s a bit of a personal challenge for me as well. Looking forward to the discussions :)
We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process
📚 reflection.ai/documents/beam…
Incredible work led by @ntumm120
Building a whole framework for quantizing linear attention models such as GDN, KDA and Mamba 🔥
The key is to leverage several computational invariances of these models in a clever way.
Check it out 👇
We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it?
It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯
Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)
Happy to share our most recent work:
An ODE-based Sampler for Discrete Diffusion Models
- we introduce a probability-flow analogue for discrete diffusion: deterministic dynamics that preserve its time marginals.
- we derive a principled discretization and efficient implementation for both MDM and UDM.
- this allows a theoretically grounded consistency distillation objective.
arxiv.org/abs/2610.40618
Joint work with @samsongvch, @ericxing, @EricMoulines, @AlainDurmus and Giacomo Zanella
We are very excited about this direction and believe that it can be pushed much further!
(1/n)
1/9
We built CoBit for language. Then we asked: what happens if we use the same architecture for images, speech and proteins?
Turn each modality into bits. Flatten. Denoise.
The surprise: how little had to change.
arxiv.org/abs/2610.14349
This will change how LLMs are trained: Parallel Language Models predict several tokens in parallel at autoregressive quality. Add a parallel decoder at pretraining, get 5.7x speedup after posttraining.
🧐 → arxiv.org/abs/2610.26693
ℹ️ → justuswill.com/plm/
Read on: 🧵 1/4
Stellar lineup at the AITHYRA Next generation AI for Biomedical Science workshop -- next Monday 2 Nov at the IMBA lecture hall in Vienna Biocenter. No registration is required.
Excited to share our work on information-theoretic scaling laws for diffusion models, with @druv_pai and @YiMaTweets!
By quantifying how much information a model learns from data, we can predict when memorization begins and how generation quality (FID) scales with training.
Can we predict memorization, generalization, and sample quality in diffusion models by measuring how much their generated samples depend on the training dataset?
In our new paper, we introduce an information-theoretic perspective and uncover a predictive scaling law. 🧵
We generated 458,000 RL environments to train an agent on stateful, multi-step on-device tasks. We share in our preprint how we decided along which axes to enlargen our RL env corpus, via data composition scaling laws.
📄 arxiv.org/abs/2610.37648
🧵1/9
🚨 New paper alert!
Despite recent advances in reward fine-tuning for diffusion and flow models, achieving high rewards remains challenging without sacrificing sample diversity or substantially drifting away from the pretrained distribution.
We address this with DFR: Distributional Functional Rewards, a framework that balances reward optimization with preserving desirable properties of the generated distribution! 🧵
🔥 Diffusion LMs promise what autoregressive models can’t: fixing their mistakes.
But new ones often change the noise and the sampler at once. Which matters?
We tested it on the hypersphere. Noise: 0.8 points. Sampler: 25.
We introduce Spherical Brownian Diffusion 🌐 👇🧵