
We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.
✓Using interpretability to understand, design, and align AI.

We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.

Read the full post and explore examples of the communities we found: goodfire.com/research/neuron…

This adds to the evidence that neural networks already contain interpretable structure if we know how to look for it. We can now find communities at scale, which agents can then investigate in depth—another tool on the path towards a fuller understanding of neural networks.

We discover these communities without specifying what they do in advance. Over millions of prompts, we identify influential neurons via gradient-based attribution, connect neurons often attributed together in a graph, and do community detection. We then test for necessity and sufficiency and see what behaviors they support.

These communities are both necessary and sufficient for their tasks. Ablating one substantially changes the model’s predictions on its task, whereas ablating everything else in its layers (besides the community and 17 always-on neurons) largely preserves the model’s predictions.

Recent work suggests individual neurons are surprisingly interpretable. But how do they work together? We found one example a few months ago: an "addition module" in Llama 8B made of just 28 neurons. Our new unsupervised method finds thousands of similar "neuron communities".

New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:

incredibly excited to be partnering with @OpenAI on embedded assessments! we'll be embedding interpretability researchers who will use the @GoodfireAI platform to monitor and debug training runs to improve safety and alignment our primary audience for our assessments are internal OpenAI researchers, but we'll publish our findings as the collaboration progresses

We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.x.com/GoodfireAI/status/2120…

Investigating neurons could have revealed that they were intractable objects of study. Instead, we find that they are more interpretable than we previously thought. Read the first post in our technical series on neuron interpretability here: goodfire.com/research/neuron…

We deeply investigated thousands of neurons in this work. What did they look like? We are releasing descriptions for 1,500 neurons in Qwen3.6 35B-A3B. Explore here: goodfire.com/research/public…

We also found that scaling agent-driven investigations reliably improves our explanations, measured by simulation scores. And each neuron we describe offers gives us clues to the next, like a crossword. Our understanding of the network compounds.

Neurons are predictable. We can generate short, natural language descriptions of neurons' roles. Providing the description to a judge model lets us predict when the neuron will fire, simulate its behavior, and activate it more strongly than anything in our corpus.

We can interpret a neuron's distinct roles by examining what function it plays in the network. If a neuron responds to both oral-health and marine-ecology, we can separate those roles by understanding the upstream neurons that activate it and the downstream readers that respond.

Neurons have relatively few computational roles, and we can identify what they are. We find that many neuron poles (positive or negative side of a neuron) have just one role. When a pole has multiple roles, they’re few in number and easy to find.

Neurons exert most of their influence on the model when they fire most strongly. That is also when they are most interpretable. Ablating the weakest 92% of activations has only modest effects on capabilities, and light fine-tuning fully recovers benchmark performance.