Frontier labs like MAI and Seed are exploring latent feedback: fusing the final layer's latent with the next token embedding as the model input.
We think the final block is the wrong one. We should read a deep intermediate block instead. Lower loss at same params & FLOPs
1/ We taught a team of Inkling 2 Small agents to work together, communicate, and delegate work, without predefined roles on hard chess puzzles. Using our simple multi-agent RL recipe, we are able to turn pass@k into pass@1.
Work with @emilianogl and @GabeVisotsky
🧵
New paper w/ Yihe Dong, @DimitrisPapail & @ShivamGarg91462: Perturbing Inputs Improves Pretraining Distillation.
In pretraining distillation, a small student LM learns to match a larger teacher’s next-token predictions on the training text. We change one thing: for the first part of training, both models read a perturbed version of the text (for example, with noise added to the token inputs), and the student matches the teacher’s predictions on that. Then we switch back to clean text.
With the same steps and token budget, this beats standard distillation, and it also works well in data-constrained, multi-epoch settings. 🧵
I finished training JEM for 100 epochs, and the results are actually better than I expected. It's only 1 point away from a fine-tuned DINOv3 (which has seen WAY more data). I'm pretty sure it could beat it with more training.
It took a bit more than 20 H100 hours to train for 100 epochs on the dataset (~250k training images), and as you can see, my LeJEPA is still undertuned (Claude is currently working on it to get a better baseline).
The main problem, though, is with rare classes, but that's expected. It might be a sampling issue. For example, I could continue training while oversampling rare classes, but that's not really what I want.
All plants kind of look the same, so I'm wondering if we could make the algorithm better at capturing fine-grained details and the patch-level loss already helps a lot imo. But tbh, the dataset is so hard that I'm not even sure there's much room for improvement on rare classes.
I'm running a small experiment to see how they compare for species classification.
I'm training both methods from scratch on a fairly small dataset, Pl@ntNet-300K (github.com/plantnet/PlantNet…): 244k training images of 1,000 plant species with a heavy long tail. I want to see which SSL method best separates rare classes.
For LeJEPA I used the official minimal implementation (MINIMAL.md). For JEM I reimplemented the paper myself and try to match its settings as closely as I could. I use opus and astra to triple check but there's still a possibility that I did something wrong
I used a ViT-B/16 backbone as the best dataset baseline for now is a fine-tuned DinoV3 ViT-B. So far JEM beats LeJEPA but it's looking too good to be true and I'm pretty sure my LeJEPA is under-tuned. But it's still better than nothing I guess, I'll have to retrain a new LeJEPA
Also comparing at equal epochs is unfair because JEM uses ~2x more compute per image (due to teacher pass). But JEM is still ahead at equal compute: LeJEPA's 100 epochs took ~23h on one RTX PRO 6000, the same as ~40 epochs of JEM, and at that point JEM's linear probe is already higher (76.6% vs 73.1% top-1). On rare species (<20 training images) they're close at equal compute (10.1% vs 9.2%).
Furthermore this task probably suits JEM better due to its patch-level objective that helps capture the small details separating species.
For reference, the released Pl@ntNet-300K-v2 ViT-B checkpoint (DINOv3 pre-trained on 1.7B images, then fine-tuned) reaches 84.5% top-1, but only ~32% on those rare species.
So it's not the end of the story, I'll continue the experiments and see if the results change with better tuning. For rare species I obvly don't expected any of them to beat a fine tuned dinoV3 as it has a better feature space, but I'm curious to see how far SSL can get from scratch with this "little" data.
how should multimodal models encode visual input?
→ pretrained encoder?
→ encoder-free?
→ before the LLM?
published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters
only one of them does
🧵 [1/n]
We tested agents on judging 2026 penalty. The best tie the referee with VAR for around 64% acc.
(not super seriously 😜) introducing Penalty Bench ⚽️ Can an AI Agent Referee?
this is a new tiny bench series made by Opus with Carbon in just 2 days.
wenhaochai.com/blogs/penalty…
I still find it crazy how silly simple Lazarus is:
1. find weak neuron
2. set in-weights to "trigger is present" direction
3. SiLU forces ~0 if not
4. Set out-weights to the unembedding of token you want screamed in the residual
5. Goes straight to logits → that token wins
1/ As agentic tasks grow longer and rollouts become more expensive, the choice between critic-based PPO and critic-free GRPO becomes increasingly important.
Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias from bootstrapping.
Our new research @ScaleAILabs compares PPO and GRPO under a shared actor objective across 1,200+ training runs in control tasks, short-horizon math, and long-horizon terminal agents. We studied how three factors influence this tradeoff: batch size, critic reliance, and task complexity. 🧵
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Was reading the score-centering RL paper to add some notes on it the my book, and they have a great table in the appendix on the different importance sampling based train-inference mismatch methods in RL. These have been the biggest focus of RL algorithm research since the GRPO takeoff, imo.
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
You can efficiently estimate calibration errors without binning. With @BachFrancis@JordanMich45361@DHolzmueller and @Eugene_Berta . The catch is to write the calibration error as a drop in risk for the right loss: no bins, a lower bound, and over/under-confidence for free.
This paper went completely under the radar!
Meta researchers have developed a novel and simple self-supervised learning framework competitive with DINOv2 and DINOv3!
"Scalable Patch-Level Self-Supervised Learning"
The authors introduce JEM, a principled student-teacher method that learns to align patch representations across different views. Additionally, it incorporates explicit regularization to ensure representations are informative and prevent structural collapse.
The framework outperforms DINOv2 in a controlled setting on dense tasks, and even beats the DINOv3 checkpoint despite it being trained on 12x less data.
Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.
StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
tinker-docs.thinkingmachines…