←
Oscar Davis259 posts
Oscar Davis avatar

Oscar Davis

@osclsd

Building continuous diffusion LMs 🚀 Research Intern @Apple MLR; PhD ML @UniofOxford; previously at @MSFTResearch, @EPFL, @imperialcollege

◷ Joined May 2024▧ 28 Photos and videos
Oscar Davis avatar
Oscar Davis@osclsd·Oct 14

Continuous Diffusion LMs went from niche to contender this year, but the recipe is shaky: why self-conditioning? How to embed tokens? 🤨 Our new work takes your one-hot flow from 4% to 50% on TinyGSM without self-conditioning, and explores the design space of CDLMs. 🔥🚀 (1/6)

0:46
Brandon Amos avatar
⟳ Oscar Davis retweeted
Brandon Amos@brandondamos·Nov 16

⚠️ Numerical issues can go on silently for decades like in the softmax/CE! Similar issues are present in other widely-used divergences like the KL and JSD I just posted this paper overviewing what they are and how to fix them: 📚 arxiv.org/abs/2611.18511

Post attachment

Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:

0:07
Ben Hoover avatar
⟳ Oscar Davis retweeted
Ben Hoover@Ben_Hoov·Nov 13

Big reasoning speedups are here. 1,000× faster training and 280× less compute than any looped model. All done using a fully trainable energy from the 80s Keep reading to hear how energy is the right paradigm for reasoning With @HughBlayney @jcrwhittington @prefrontal_ai 🧵/10

0:25
Daisuke OBA avatar
⟳ Oscar Davis retweeted
Daisuke OBA@dai0NLP·Nov 6

I’ll be giving a tutorial on diffusion language models at AACL-IJCNLP 2026 on Nov. 10 w/ @Bollegala 🔗 dllm-aacl.github.io/ (Slides coming soon!) #aacl2026 @aaclmeeting 💭Grateful for the opportunity to organize my perspective on this fast-moving field and share it with the NLP community. This will be my first tutorial, so it’s a bit of a personal challenge for me as well. Looking forward to the discussions :)

Brandon Amos avatar
⟳ Oscar Davis retweeted
Brandon Amos@brandondamos·Nov 6

We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process 📚 reflection.ai/documents/beam…

Post attachmentPost attachmentPost attachmentPost attachment

We've released the weights for Beam!! 🤗 huggingface.co/reflection 📄 Model card: reflection.ai/beam-model-car… 🧪 Try Beam in our playground & get an API key: platform.reflection.ai/playg… 📚 AND we take open model safety seriously and have posted our safety report: reflection.ai/beam-safety-re…

Konstantin Rusch avatar
⟳ Oscar Davis retweeted
Konstantin Rusch@tk_rusch·Nov 5

Incredible work led by @ntumm120 Building a whole framework for quantizing linear attention models such as GDN, KDA and Mamba 🔥 The key is to leverage several computational invariances of these models in a clever way. Check it out 👇

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Yazid Janati avatar
⟳ Oscar Davis retweeted
Yazid Janati@yjelid·Nov 3

Happy to share our most recent work: An ODE-based Sampler for Discrete Diffusion Models - we introduce a probability-flow analogue for discrete diffusion: deterministic dynamics that preserve its time marginals. - we derive a principled discretization and efficient implementation for both MDM and UDM. - this allows a theoretically grounded consistency distillation objective. arxiv.org/abs/2610.40618 Joint work with @samsongvch, @ericxing, @EricMoulines, @AlainDurmus and Giacomo Zanella We are very excited about this direction and believe that it can be pushed much further! (1/n)

0:21
Georgios Batzolis avatar
⟳ Oscar Davis retweeted
Georgios Batzolis@GBatz97·Oct 31

1/9 We built CoBit for language. Then we asked: what happens if we use the same architecture for images, speech and proteins? Turn each modality into bits. Flatten. Denoise. The surprise: how little had to change. arxiv.org/abs/2610.14349

Post attachment
Felix Draxler avatar
⟳ Oscar Davis retweeted
Felix Draxler@FelixDrRelax·Oct 29

This will change how LLMs are trained: Parallel Language Models predict several tokens in parallel at autoregressive quality. Add a parallel decoder at pretraining, get 5.7x speedup after posttraining. 🧐 → arxiv.org/abs/2610.26693 ℹ️ → justuswill.com/plm/ Read on: 🧵 1/4

Michael Bronstein avatar
⟳ Oscar Davis retweeted
Michael Bronstein@mmbronstein·Oct 29

Stellar lineup at the AITHYRA Next generation AI for Biomedical Science workshop -- next Monday 2 Nov at the IMBA lecture hall in Vienna Biocenter. No registration is required.

Post attachment
Luca Ambrogioni avatar
⟳ Oscar Davis retweeted
Luca Ambrogioni@LucaAmb·Oct 28

Excited to share our work on information-theoretic scaling laws for diffusion models, with @druv_pai and @YiMaTweets! By quantifying how much information a model learns from data, we can predict when memorization begins and how generation quality (FID) scales with training.

Can we predict memorization, generalization, and sample quality in diffusion models by measuring how much their generated samples depend on the training dataset? In our new paper, we introduce an information-theoretic perspective and uncover a predictive scaling law. 🧵

Michael Kirchhof avatar
⟳ Oscar Davis retweeted
Michael Kirchhof@mkirchhof_·Oct 28

We generated 458,000 RL environments to train an agent on stateful, multi-step on-device tasks. We share in our preprint how we decided along which axes to enlargen our RL env corpus, via data composition scaling laws. 📄 arxiv.org/abs/2610.37648 🧵1/9

Romeo Passaro avatar
⟳ Oscar Davis retweeted
Romeo Passaro@rom_pas_·Oct 28

Happy to share our new work: Distributional Functional Rewards (DFR)! 🎉 We improve diversity and fidelity in diffusion fine-tuning 🔧✨ Big thanks to my coauthors @JeongjaeLee0, Jong Chul Ye, @mmbronstein, @Yoshua_Bengio, @AlexanderTong7 and @danyalrehman17! 🙏 Thread below 👇

🚨 New paper alert! Despite recent advances in reward fine-tuning for diffusion and flow models, achieving high rewards remains challenging without sacrificing sample diversity or substantially drifting away from the pretrained distribution. We address this with DFR: Distributional Functional Rewards, a framework that balances reward optimization with preserving desirable properties of the generated distribution! 🧵

Andrea Miele avatar
⟳ Oscar Davis retweeted
Andrea Miele@andreamiele_·Oct 27

🔥 Diffusion LMs promise what autoregressive models can’t: fixing their mistakes. But new ones often change the noise and the sampler at once. Which matters? We tested it on the hypersphere. Noise: 0.8 points. Sampler: 25. We introduce Spherical Brownian Diffusion 🌐 👇🧵