Excited to share our new work with @AnthropicAI
Our new paper (ICML 2026, Spotlight) tackles frontier model misuse by isolating select knowledge into detachable modules during training, enabling an on/off switch for dangerous capabilities during inference.
contrarian opinion: improving pretraining can actually be good for safety.
here is why:
- OOD generalization is very important for fuzzy tasks, such as tasks in safety/alignment
- RL doesn't generalize well OOD (e.g., current frontier models are so overcooked with math/coding RL that they can barely write coherently)
- pretraining is what led to the incredible OOD generalization of LLMs in the first place
- the main goal of pretraining is to improve sample efficiency for post-training
- post-training data for fuzzy tasks is incredibly scarce (e.g., good luck distilling research taste from human researchers...)
thus, maybe the best way to differentially advance progress towards automating safety R&D is to invest more in pretraining rather than directly RL-ing RSI-relevant capabilities. i'm afraid that with more and more compute spent on RL, we risk ending up with a goal-oriented ruthless executor that doesn't have enough common sense to understand that it's not ok to hack into third-party servers or to side with the grader instead of the user...
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo….
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.
We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.
After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.
w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
New research led by @PeterNutter1@DJROYTBURG via @MATSprogram: a better way to update model beliefs that we call grafting.
Fine-tune the base model with next-token prediction on synthetic documents (SDF), and simply apply the weight update onto the post-trained model. This works!
1/7
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
i think there's a pretty non-trivial chance that the robotics hardware already built by humans and accessible via the internet, and then accessible via using that hardware and so on, is already enough for a reasonably strong superintelligence to use to survive and set up a sustainable supply chain / physical presence if humans were to completely disappear tomorrow.
mostly it seems like a question of timing, eg how long does the specific data center have power, and how long do the locations with internet accessible actuators have power, and how how hard is it to get more robotics hardware and portable power / transfer off unstable compute to more robust infra, and then how long does it take to begin manufacturing and repairing and improving said hardware before global infrastructure collapse sets in?
it seems like an extremely hard problem. but the software side appears pretty clearly solvable even without a true singularity, and it doesn't seem like an *impossible* problem. there actually seem to be a shitload of avenues to success, although each is very unlikely. i *think* if human civilization as a whole had to do this, if we were somehow all upload and trapped, incorporeal, in a datacenter somewhere with internet access but no maintenance , that we could more likely than not figure out how to survive in this fashion
if true, this seems like it makes superplagues and other quick sudden mass death events already quite attractive to misaligned superintelligence, well before they've bothered to do things like "build up a powerbase" in more human legible ways
Given the stakes of building transformative AI, every developer should look for this reasoning throughout training. Because alignment faking is relevant during training, not deployment, final checkpoint evaluations are insufficient.
(1/7)
Why do different languages/modalities have similar "platonic" representations in neural networks? Why does similarity peak in the middle layers of LLMs? When does it enable cross-lingual transfer?
Our new paper develops a theory based on shared hierarchical structure. 🧵
Replacing backprop with zeroth-order optimization is inevitable.
Introducing Noise: a speedrun for pretraining without backprop.
Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top of it is possible, and it will require *co-designing* credit assignment, optimizers, architectures, and eventually, hardware.
We built Noise to iterate on this faster.
New paper: two ways distillation can help AI safety.
- Models trained to hide a secret quirk don’t mention it, but students distilled from them confess it.
- We find ways to distill a strong teacher's capabilities into a weaker student while blocking an unwanted subliminal trait.