1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token.
Can we scale up this single stream computation?
Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows).
RNN 🤝 Transformers 🧵
Reasoning got loops. Vision didn’t 👀
Introducing AdaLoop: giving vision its loop 🔁. A pretrained VLM looks at the image again and again, steered by the question, through its own frozen visual layers 🧵
OPAL also improves 0th-order optimization! A clean new piece of evidence found by Robert that online reactivity can power all types of optimizers.
“
There is an interesting moment at update 243: The record switches its gradient estimator (Boost → Cut), and the gradients change scale. All three agents had tuned their optimizers on Boost's gradients, and they lost ground right after the switch, up to 0.03 in training loss. OPAL noticed, rescaled the LM head's Adam statistics to the new gradients, and kept adjusting. By update ~620, every run was ahead of the record's optimizer again, and the informed runs stayed ahead to the end.
“
I was wondering if OPAL could help the optimizer for the 0-order noise val track.
So I took the (old 10M token) Noise benchmark's best validation-loss record (#14, 3.676) and wrapped its optimizer with OPAL. Every 50 updates, gpt-6.1-sol reads the loss and its own diagnostics, rewrites the update rule in Python, and tests it before it goes live.
Results (seed 0):
- record: 3.6758
- no agent (rerun): 3.6773
- agent, neutral prompt: 3.6791
- agent, prompt informed by the record: 3.6695
- same, xhigh reasoning: 3.6656 (−0.010 vs the record)
The OPAL system prompt mattered: telling the agent what its gradients are, how the record's optimizer came about, and how the end-of-run weight merge works turned no gain into a clear one.
There is an interesting moment at update 243: The record switches its gradient estimator (Boost → Cut), and the gradients change scale. All three agents had tuned their optimizers on Boost's gradients, and they lost ground right after the switch, up to 0.03 in training loss. OPAL noticed, rescaled the LM head's Adam statistics to the new gradients, and kept adjusting. By update ~620, every run was ahead of the record's optimizer again, and the informed runs stayed ahead to the end.
What the best agent changed, keeping the record's ANVIL II update for the weight matrices:
- per-layer rates for the residual mixing scalars, which it halved at the start
- matrix rate −10%, +20% on two query matrices, −15% on the value matrices
- longer momentum averaging (higher β1) for the scalars, head and gates
- the head's Adam rescale at the estimator switch
This is, of course, only anecdotal, and a minor improvement. Still, demonstrates that OPAL doesn't diverge and can mine small gains.
Intuitively, online agent-guided adaptation makes the most sense when hyperparameters/configurations haven't been tuned to death.
Sorry again to our infra engineers out there 😶 but changing architectures mid-training (e.g. adding and removing sub-modules) to respond to the fine-grained dynamics inside the network sounds pretty fun to me...
I love the research direction!
You might be interested in arxiv.org/pdf/2405.16195. It tunes the HPs, architecture, and optimizer by itself, during training.
@AlexDGoldie also worked in that direction with learned optimizers arxiv.org/pdf/2407.07082
The 2 focus on autoRL.
Indeed, it slows down training a bit! Section 4 of the blog discusses this, and I’ve attached some additional equal-time comparison plots. You’re right that, since I make agents react infrequently in my experiments, the overhead per run is manageable: at most 2 hours and $6–$15 in tokens. In general, the relative overhead is lower for larger-scale training runs.
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
YES! i believe AGI can check every data every neuron every hyper-param in optimier per every step, and this is what the next 10x we can get from pretraining!
Agent cost: In our default settings, Opus 5.5 high costs $5 to $15 per run. We limit the reaction time to 100 at max; this can be adjusted at will to control budgets or perform more fine-grained control.
Send our pip package (OPAL, OPtimizer with an Agent in the Loop) to your agent and ask it to equip your training with reactive optimizers!
💻 github.com/xingjian-bai/opal
Joint work with Jia-Hao Liang, @jayden_teoh_ and @vincesitzmann.
In the past decade, deep learning models have come to possess unreasonably effective power through training. Yet the very heart of their making, the optimizer, is still mechanistic.
Our post is an attempt to breathe the magic that training has given us into the core of training itself, forming a loop that we hope will create more capability in the future!
8/9
Suppressing spikes and overfitting is something human experts can still do, only less conveniently and at a higher cost. As agents get smarter, with hundreds of papers in their memory, thousands of tokens in their chain of thought and millions of training signals in their context, we are convinced reactive optimizers will invent new mechanisms beyond what human experts could invent by hand. Reactiveness can also go beyond gradient-based methods to zeroth-order (like Dust @industriaalist), language-based, and more.
7/9
Reactive optimizers expand what an optimizer can do, with a reasoner in the closed loop. They autonomously build their own training recipes, each adapted to its run: multi-stage schedules for learning rate, clipping, momentum, and more. One switches to per-module clipping after tracing 17 of 18 gradient spikes to a single module; another keeps one matrix's moments in FP32 after deciding rounding would eat its updates. More details are in the blog: the decisions they make are highly specialized.
6/9
Can an optimizer overcome overfitting by itself? We train a ResNet-18 on CIFAR-10 with 10% wrong labels. Trained straight through, AdamW ends up learning the assigned wrong labels.
"The model has started to memorise its training images, and dev loss is climbing fast." Noted by the reactive optimizer at epoch 24. Then it avoids overfitting by itself, navigating by dev-set signals, raising its learning rate, and averaging its weights.
5/9
We simulate the scenario above in language and image tasks, where no recipe has been optimally tuned for the setting: modded-nanogpt taken to new languages or to a quarter of the batch, Goldfish models on new languages, and LightningDiT and RAEv2 on ImageNet. In all of them, the reactive optimizer beats its starting recipe and matches or beats the best recipe one could pick off the shelf.
4/9