Tinker’s Dimuon is Muon with delayed diagonal conditioning, Polar Express and a rank-dependent update norm. I recovered its update rule through the API, borrowing @mrtnm’s approach to StableReinforce. Here’s how 🧵
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at tinker-docs.thinkingmachines…
score centering + masked importance sampling is now the default in TRL's async GRPO Trainer!
Across various staleness and sampler quantization settings, the stability is pretty wild 👇
score centering + masked importance sampling is now the default in TRL's async GRPO Trainer!
Across various staleness and sampler quantization settings, the stability is pretty wild 👇
On a side note, I'm thrilled Thinky has found Score Centering to be extremely stable! From their docs: "In our internal post-training, this loss function has led to extremely stable training, even under highly off-policy async RL and low precision numerics."
So we have a hypothesis: StableReinforce = Reinforce + Score Centering, where the expected score is computed over 8 randomly sampled tokens. We can easily verify that it is true: we implement Reinforce + SC as a custom loss in Tinker, do an optimizer step, and recover gradients that are within numeric precision of StableReinforce across all layers.
Looking at a single-token update, surprisingly only 8 logits have a non-zero gradient. In contrast, a standard Reinforce loss would put a non-zero gradient on all logits.
This observation basically reveals the objective. In Score Centering, we subtract the expected score under the sampler. Since computing the exact expectation is expensive, we have to resort to an approximation. In our paper, we propose a top-k approximation, which is deterministic but slightly biased. It seems that Thinky instead decided to compute the expected score by randomly drawing 8 tokens from the sampler, which is unbiased but higher variance.
Every token’s score has the same dense part (−p, from the softmax), so subtracting the average score over the 8 correction tokens cancels it exactly.
We train a rank-1 LoRA adapter on top of Qwen3.5-4B. Since the LoRA adapter (B · A) for the unembedding matrix is rank-1, the gradient w.r.t. B is equal to the gradient w.r.t. logits (up to a scaling factor). In other words, we can recover the exact logits gradient!
Tinker does not expose what the loss function and "stability info" are. However, one thing we can do is take an optimizer step on the loss function, download the updated weights, recover the raw gradients, and reverse-engineer what loss function would generate the same gradients.
Tinker doesn't support SGD, so we use Adam with ϵ=10,000. This value of ϵ dominates Adam's second moment, so Δw ≈ −lr·g/ϵ, i.e. we can recover the raw gradients from the update.
Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.
StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
tinker-docs.thinkingmachines…
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
We reproduced this in TRL's AsyncGRPO 👇
Qwen-1.5B, vLLM in FP8 (weights + KV) and trainer in bf16:
• plain GRPO collapses after ~150 steps, reward 0.56 → 0.10
• ratio caps and masked IS alone don't save it
• score centering + MIS trains like a bf16 sampler (0.72)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
A cool application of score centering: running RL on heterogeneous hardware
It uses the Miles implementation of score centering, which supports top-p sampling.
Applying score centering to top-p sampling is a generalization of the same "mask replay" correction that Cognition and DeepSeek use for top-p training (under zero TIM, these corrections are equivalent). Another benefit is that under top-p sampling, it is feasible to run exact score centering (which is infeasible with full sampling).
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]
RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem.
We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement!
🧵 [1/6]