New paper: World Modeling in Transformers
with @matthieu_queloz@andrenfreitas@MATSprogram
Can AI models recover world models just from data? We find a faithful internal map and more inside of taxiGPT! We trace failures to how that the map is stored in superposition. 🧵(1/11)
New neurons research: introducing Neuron Communities.
We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date.
Work led by research fellow @_igorshilov:
🚨 MIT proved you can delete 90% of a neural network without losing accuracy.
The “lottery ticket hypothesis” found that some networks contain smaller subnetworks that can match the original model’s accuracy.
These “winning tickets” can be trained from their original starting weights. The finding does not apply to every model.
The challenge was finding them.
The original method required training the full network first, then pruning it and retraining the remaining connections.
That extra work made the approach harder to use in production.
Hardware support offers another way to benefit from sparsity.
NVIDIA Ampere and newer GPUs support a specific 2:4 pattern, where two weights out of every four are zero.
Compatible operations can use less memory bandwidth and reach up to twice the compute throughput.
But 2:4 sparsity removes 50% of those weights, not 90%. It also does not guarantee twice the speed for an entire model or zero accuracy loss.
Training methods and software support help turn fewer weights into practical savings. Results still depend on the model, hardware and pruning pattern.
The lesson is promising: some networks can achieve similar accuracy with far fewer connections.
Finding those connections and running them efficiently remain separate challenges.
New GovAI post: I argue Frontier AI companies should publish their third-party evaluation contracts. This would show the conditions evaluators work under, make results easier to interpret and compare, and create incentives to improve evaluation arrangements.
More in thread 🧵
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
New preprint:
Cross-modal steering vectors transfer zero-shot in both directions 🖼️ ↔️ 📝
From text generation to image generation and vice-versa
Exciting evidence for a shared conceptual space!
Fantastic effort led by Donghao Zeng and awesome co-authors @nsubramani23 & @TamarRottShaham
Details 🧵👇
You can turn Qwen3-8B into an owl by training only one random neuron! 🦉
Performance on owliness evals approaches full SFT. Compare SFT vs one-neuron eval responses side by side here: maxsloef.github.io/owl-perso…
You can turn Qwen3-8B into an owl by training only one random neuron! 🦉
Performance on owliness evals approaches full SFT. Compare SFT vs one-neuron eval responses side by side here: maxsloef.github.io/owl-perso…
check out this intuitive video on why interpretable directions in llms look non-gaussian, how ica finds them in minutes per layer, and the evaluation bug that made ica look weak for 3 years
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
"three reasons why AI isn't conscious." really?
"consciousness is not intelligence": not a reason why AI isn't conscious, just rebuts a bad argument for AI consciousness (AI: intelligent therefore conscious).
"consciousness is substrate-dependent"/"noncomputational": would be a reason if true, but truth needs to be established.
more accurate title: "three reasons why AI might not be conscious".
x.com/Nature/status/21121504…
What is an LLM?
Is it an incomprehensible black box, a next-word-predictor, a silicon genius?
To answer this, we don't have to only guess, we can open it up and look inside!
It was an honor to chat with @lennysan about what we learned doing this, some takeaways in 🧵
"To make them is not to understand them."
Emmanuel Ameisen (@mlpowered) has spent the last 3 years doing interpretability research at @AnthropicAI, looking inside LLMs like Claude to figure out what they're actually doing. As he puts it, models are "grown," not built.
We discuss:
🔸 His discovery that models plan many tokens ahead
🔸 Why anthropomorphizing AI can actually be the right approach
🔸 What the "J-space" reveals about how models think
🔸 How interpretability connects to AI alignment and safety
Listen now 👇
youtu.be/JHo-aiePjoM
Still, I don’t think this is a paradigm shift for mech interp. We’ll still need diff-means, probes and SAEs to find features (in particular for the res stream). But I’m excited to see more attribution graphs built directly on MLP neurons! 17/
Overall, Nick gives evidence that there are fewer geometry tricks than we thought, at least for MLP neurons. More widely, this points toward superposition being driven more by structure than by geometry tricks. If true, latent spaces are indeed cleaner than prev. expected. 16/
Things might be different in the residual stream, though. Every att head and MLP reads from and writes to it, so it may need to carry a lot of information. So space-saving superposition and geometry tricks could be more common there. 15/