
Beyond Score Centering: what else can we fix in async RL? 🧵 Score centering (SC) is the hot new trick for off-policy LLM RL. We stress-tested it under real async staleness (every batch up to 127 steps old) and found what it fixes, what it doesn't, and two cheap wins on top 🔍 What SC actually fixes • Without SC, a trajectory's gradient norm grows ~linearly with its age: at age 64 it's 3.2× an on-policy one. With SC: 1.5× • The culprit is a hidden drift term, an uncontrolled −A·KL(μ‖π) baked into the policy gradient. SC removes it ⚠️ What SC + IS gets wrong • SC then IS → adds a new drift • IS then SC → a no-op without truncation; with truncation it centers on a distribution with no guarantee of being closer to π • So "IS and SC are orthogonal" (from the SC paper) doesn't hold Two wins on top of SC 1️⃣ State-level KL mask: drop states where KL(μ‖π) > τ, instead of masking by token-level ratios. It depends only on the state, so SC's baseline property survives. At age 127: 3/3 seeds climb steadily, while TIS+SC and MIS+SC spike and crash 2️⃣ Random buffer sampling, a free lunch from the system: same buffer, sample at random instead of FIFO. Mean age 127 → 52, better for every method at equal wall-clock Recipe = SC + KL penalty + KL mask. At lr 4e-2 (w/ SGD), MIS+SC and DPPO collapse and TIS+SC swings, while every recipe seed trains and learns faster early 🤔 So far, we’ve only tested on Countdown with Qwen3-0.6B (wants to scale it up!). When staleness is low, all the fixes perform similarly, but the differences become clearer as staleness increases. We also haven’t compared against sequence-level masking (which might already be enough!) or isolated kernel mismatch from async lag. Both are worth looking into next. #LLM #RL #AsyncRL #ReinforcementLearning




















