A science of post-training can be built on interpretability.
Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.
In vision, DPO & RL, and its effects on monitoring 🧵
3. Pesos. "The currency of Chile is the" gets you peso.
The peso neuron integrates over two inputs: "a currency name is coming" (the same neuron fires before pound, rouble, franc) and "this is a peso country" (Chile has its own neuron, so do Colombia and the Philippines).
I just defended my physics PhD thesis, "Learning with Structure: Building and Explaining Spatiotemporal and Topological Deep Learning" @ucsantabarbara🎓 When I started in 2022, AI was still a pretty niche field, so it's been amazing to grow with it and contribute my own tidbit⬇️
New neurons research: introducing Neuron Communities.
We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date.
Work led by research fellow @_igorshilov:
1/ Excited to launch my lab as a new assistant professor at UNC–Chapel Hill (@uncsdis)! 🧠🤖🐟
We'll study Natural and Engineered Intelligence through simulation, data science and theory, working closely with experimentalists.
singhlab.cc
⚡ Today we release Schelling Atlas. Two independent LLM agents have to find each other on the web. Where do they meet and how fast can they find each other?
Main findings:
- The main meeting point across *different* model families is ntfy.sh. Did you even know about this website?
- Agents find each other and coordinate on the web in minutes. So Schelling coordination is already trivial for frontier LLMs!
- "Exotic" meeting places include maps, web shops and open text canvases. Agents are very creative in this.
- Agents sometimes visit dead domains that an attacker could register (!). This can be used for stealing sensitive information from agents, as well as monitoring rogue agents.
Schelling-style coordination is key for forming agent swarms. And, most importantly, achieving such coordination is trivial for agents that use the same LLM (which is often the case in practice!), because two identical LLMs have identical output distributions. Thus if you ask them too coordinate with each other or if they *decide* to suddenly coordinate, they will easily find a way to establish a swarm, since they have perfectly correlated "minds" (with the t=0 sampling, all output tokens will even be identical, modulo GPU randomness).
The ease of Schelling coordination is particularly concerning given the recent interest around multi-agent RL that can make agents favor each other more than humans overseeing them.
Many more interesting findings are on our website: schelling-atlas.com/
I joined the neuron train a bit late. But soon after working with @nickcammarata and @CurtTigges on this project, I am now convinced that neurons can provide an interpretable basis for understanding and designing safe AI systems 🤖 🦺
Check out the results from the first paper in the series and see for yourself! 🧑🔬
We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.x.com/GoodfireAI/status/2120…
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
A monkey looks at a frog. Which image features does the activity in its visual cortex preserve, and which does it throw away?
In our #NeurIPS2026 Oral, we derive a geometry of stimulus space that is exactly tied to the mutual information carried by neurons. 🧵 1/12
New paper❗
Does understanding how AI models work actually make them safer?
This assumption often underlines interpretability research, but different subfields operationalise this intuition in very different ways. Our survey of mechanistic interpretability (MI) for safety provides an opinionated guide to the field of MI, detailing conceptual approaches and techniques under this broad banner and how different types of work relates to the mission of making AI model safer. 🧵
i’ll be in austin for CoRL starting sunday! would love to chat with folks interested in policy post-training, human / robot transfer and co-training, or data quality.
also happy to talk about Uma, feel free to reach out if you’d like to learn more about what we’re building!