
We found a problem with on-policy self-distillation. The student makes mistakes early in a rollout, and the teacher has to continue from that flawed prefix, so its learning signal gets a lot weaker. So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot. We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more! Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.













