←
Zhe Wang25 posts
Zhe Wang avatar

Zhe Wang

@zhexwang

Frontier AI @GoogleDeepMind. Generative modelling, Robotics and AI4Science.

◷ Joined December 2010▧ 6 Photos and videos
Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(1/6) Can Autoregressive Models (ARMs) go beyond fixed or random generation orders and learn the optimal order to generate new samples? We believe so! We are excited to present our paper, “Learning-Order Autoregressive Models with Application to Molecular Graph Generation,” at ICML 2025 in Vancouver on July 18th. Feel free to drop by! In our new paper, we introduce Learning-Order Autoregressive Models (LO-ARMs), which are trained to find a consistent, context-dependent generation order on their own. Our model matches and surpasses state-of-the-art results on two major molecular graph generation tasks. Read our paper here: arxiv.org/abs/2503.05979 Joint work with @thjashin, Nicolas Heess, @ArthurGretton, Michalis K. Titsias More in 🧵👇

Post attachment
Sander Dieleman avatar
⟳ Zhe Wang retweeted
Sander Dieleman@sedielem·Aug 7

Excellent talk by @thjashin: - What separates autoregression and diffusion? - Bridging the gap between continuous and masked discrete diffusion via loss reweighting - Insertion-based sequence generation Very clear and easy to follow! (via @mblondel_ml) youtube.com/watch?v=kMimQxIJ…

Jiaxin Shi avatar
⟳ Zhe Wang retweeted
Jiaxin Shi@thjashin·Jul 6

AR models append tokens only at the end. But what if generation could insert tokens anywhere? It's strictly more expressive, but much harder to train with standard max likelihood. We introduce a scalable variational learning method for insertion-based generative models. arxiv.org/pdf/2606.02133

Post attachment
Lester Mackey avatar
⟳ Zhe Wang retweeted
Lester Mackey@LesterMackey·Mar 20

Qiang Liu, Chris Oates, and I are writing a monograph on Probabilistic Inference and Learning with Stein’s Method, and we’d love to get your feedback on the first draft

Post attachment
Jiaxin Shi avatar
⟳ Zhe Wang retweeted
Jiaxin Shi@thjashin·4 Aug 2025

Nice writeup on CTMC and discrete diffusions

Wrote up some notes providing an introduction to discrete diffusion models, going into the theory of time-inhomogeneous CTMCs via generators/time-evolution systems. What motivated me was the sheer difficulty of finding a useful reference which laid out the theory (e.g. Kolmogorov equations etc.) in a single place. Hope these can be helpful to others.

Zoubin Ghahramani avatar
⟳ Zhe Wang retweeted
Zoubin Ghahramani@ZoubinGhahrama1·2 Aug 2025

Google DeepMind has been leading the revolution in Foundation Models for Robotics. Come join us. We are hiring!

It takes a village to build an awesome 🦾 🧠 . We are #hiring! Come transform robotics at Google DeepMind. I've received a lot of request for applied and engineering roles. Indeed we are hiring for those too! Robotics engineering: lnkd.in/gDt6CeAq Applied ML SWEs: lnkd.in/gCmY69XQ Research Scientist: lnkd.in/g47zS2Xz

Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(6/6) Finally, can LO-ARMs generate "better" molecules? Yes, they can. On both the QM9 and ZINC250k datasets, LO-ARMs match or exceed state-of-the-art (SOTA) results. Performance was evaluated across key metrics for distribution similarity and drug-likeness.

Post attachment
Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(5/6) Can LO-ARMs learn consistent and meaningful generation orders on their own? Yes — and the strategy/generation order they discover is surprisingly intuitive. On two major molecular generation tasks (QM9 and ZINC250k), our models learned a consistent, two-phase process: 1. Build the Skeleton: First, generate the chemical bonds. 2. Infill the Atoms: Then, add the atoms to the skeleton. We find that 99% of the new samples follow this generation order. Here’s an example of this in action for generating a ZINC250k molecule. We start with a blank slate where all tokens are masked, and in each step, the model adds one new piece — either an atom or a bond, until the molecule is complete.

Post attachmentPost attachment
Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(4/6) How do we train this on a large scale? The key to training our model at scale is variational inference. We design a variational order policy that lets us optimize a lower bound on the data's true likelihood, and we incorporate two crucial techniques to make this process fast and stable. First, by conditioning the policy on the full input, we can sample the entire generation order in parallel using the Gumbel Top-k trick, which dramatically speeds up training. Second, we use the REINFORCE Leave-One-Out method to reduce gradient variance, leading to more stable and reliable convergence.

Post attachment
Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(3/6) How can LO-ARMs fix the problems? A LO-ARM consists of an order policy (to choose which dimension to generate next) and a classifier (to decide its value). The key difference is that this generation order isn't handcrafted or random - it is learned purely from data and is fully context-dependent, adapting as the sample is generated.

Post attachment
Zhe Wang avatar
Zhe Wang@zhexwang·13 Jul 2025

(2/6) So, why is enabling ARMs to learn their own generation orders so important? Many problems can be modeled as next-token prediction. While text has a natural left-to-right order, other data types, like images and molecular graphs, lack an obvious canonical order sequence, making them harder to generate sequentially. Two common generation strategies are: 1. Fixed-Order (FO): Generating tokens in a set order. 2. Any-Order(AO) / Masked Diffusion: Generating tokens in random orders. We illustrate their differenes and problems in the image blew.

Post attachment