←
Arshia Soltani65 posts
Arshia Soltani avatar✓

Arshia Soltani ✓

@ckodser

MS&E PhD @ Stanford

◷ Joined August 2021▧ 3 Photos and videos
Arshia Soltani avatar
Arshia Soltani@ckodser·Nov 12

We found a problem with on-policy self-distillation. The student makes mistakes early in a rollout, and the teacher has to continue from that flawed prefix, so its learning signal gets a lot weaker. So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot. We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more! Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.

1:10
0:57
Mohammad Hajiaghayi (IG@mhajiaghayi,YT:hajiaghayi) avatar
⟳ Arshia Soltani retweeted
Mohammad Hajiaghayi (IG@mhajiaghayi,YT:hajiaghayi)@MTHajiaghayi·Nov 13

Our Theo system formalized all 22 of Robertson and Seymour's Graph Minors papers, deepest theory in combinatorics: all 838 numbered results, no extra axioms. Including graph minor theorem (Wagner's conjecture), excluded-minor structure theorem, and O(n³) k-disjoint paths algo.

Jonas Hübotter avatar
⟳ Arshia Soltani retweeted
Jonas Hübotter@jonashubotter·Nov 13

On-policy learning is wasteful. Trajectories following an early mistake often don’t recover and waste many tokens on non-useful signal. We study speculative self-distillation: sampling as on-policy as possible while correcting “stupid” mistakes early. This improves the efficiency-accuracy-forgetting frontier. Every token counts! Awesome work with @talaei_shayan, @chewing_a_gum, @ckodser, Amin Saberi, and @Azaliamirh.

0:57
François Costa avatar
⟳ Arshia Soltani retweeted
François Costa@cos_francois·Nov 12

Speculative Self-Distillation (SSD) lets AI models learn new knowledge from their own outputs, with a teacher stepping in only when predictions significantly diverge. The result: 38–69% fewer training tokens, near on-policy accuracy, and less catastrophic forgetting. Great work from @talaei_shayan 🚀

0:57
Agam Bhatia avatar
⟳ Arshia Soltani retweeted
Agam Bhatia@chewing_a_gum·Nov 12

New paper! On-policy self distillation is super wasteful. The teacher is only useful early in the rollout, and the student cannot course correct later. With SSD, we handhold the student, and see much faster convergence and lesser forgetting!

0:57
Azalia Mirhoseini avatar
⟳ Arshia Soltani retweeted
Azalia Mirhoseini@Azaliamirh·Nov 12

Introducing Speculative Self-Distillation (SSD), which improves accuracy and effiecieny and reduces forgetting in self distillation. It turns out only a few tokens in a rollout actually depend on the new knowledge. Letting the teacher generate just those tokens results in nearly matching on-policy self-distillation with 38–69% fewer training tokens, and the model forgets less. Great work co-led by @talaei_shayan, @chewing_a_gum and @ckodser, with @jonashubotter and Amin Saberi (NeurIPS 2026). Check out Shayan’s thread for more.

Post attachment
0:57
Shayan Talaei avatar
⟳ Arshia Soltani retweeted
Shayan Talaei@talaei_shayan·Nov 12

Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation. In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track. 📉 38–69% fewer training tokens than on-policy, matching its accuracy ⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A) 🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy

0:57
Arshia Soltani avatar
Arshia Soltani@ckodser·Nov 4

Training a sota model with 5 people is crazy. I wonder if reaching this level needs <50 million and 5 researchers, does ANT really worth 1.5T?

Ricursive Intelligence avatar
⟳ Arshia Soltani retweeted
Ricursive Intelligence@RicursiveAI·Nov 4

Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch. Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B. Learn more: ricursive.com/ripple

1:14
Jacky Kwok avatar
⟳ Arshia Soltani retweeted
Jacky Kwok@jackyk02·Aug 17

Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰 As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost. For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench. Try it out today: github.com/llm-as-a-verifier… More on verification scaling in my previous post.

Post attachmentPost attachment
Arshia Soltani avatar
Arshia Soltani@ckodser·Jul 23

The author just confirms the bug our system detected in the STOC 2025 paper. AI reviews are getting real and even better than the STOC-level review process.

1:31
Arshia Soltani avatar
Arshia Soltani@ckodser·Jul 3

We built an automated pipeline that translates research math into formal languages. It successfully formalized 5 STOC papers! 🚀 It turns out AI systems are incredibly capable at translating complex human reasoning into machine-verifiable code.

1:31