Synthetic Persona Pretraining: Alignment From Token Zero – our full paper is finally out. We train up to 3B models and inject synthetic morally-laden reflections into 10% of pretraining documents.
Surprisingly, intervening early really shifts the model's value priorities 🧵
If all knowledge is (in theory) understandable to humans, then shouldn't we pursue the mission of expanding human insight?
Shouldn't we open the library doors?
If all we accomplish is to create superhuman AI that people do not understand, then AI will have failed humanity.
As long as it will look look like you have to produce k papers at top conf to get any type of position (master/phd/internship/grants/...) this will continue.
A single direction in LLM activation space, applied as a steering vector with four different norms, can cause the language model to perform a different behavior at each norm.
This and more in a new blog post, Do Not Rely on the Linear Representation Hypothesis. (link below.)
So much still to discover about optimizers! 🔍 We spend a lot of time comparing pretraining loss, but the real test is what happens after fine-tuning.
More pretraining data can actually flip the Muon–AdamW ranking after fine-tuning.
Check out the thread for the details, plus some nice theoretical insights into the geometry of Muon and forgetting 🧵👇
Presented as an oral at the MOSS workshop at COLM! 🏆
Muon achieves lower loss than AdamW after pretraining, but does its advantage persist after fine-tuning?
Surprisingly, the answer depends on pretraining data scale: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning!
Rethinking pre-pretraining (PPT)
Allocating compute to synthetic PPT never outperforms pure natural-language training.
In our compute-matched settings: even when natural language is scarce, repeated epochs over a 100M-token corpus beat every synthetic curriculum. 🧵
It looks like Astra didn't natively read Qwen's weights?
As I understand it, Qwen would get stuck in a loop and not answer the question.
Astra used code to perform some standard interp techniques, found a looping direction, and ablated it from the weights.
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Just fully* resigned from JMLR. I still believe in science, but journals needs a complete rethink. AI reviews by top models are now much better than time-pressed human ones. OTOH, one-time gate keeping (accept / reject) doesn't make sense. The literature is going to be moving faster than ever. We need to think about it as a distributed data structure, and start over.
This "persona neuron" stuff seems like it's mostly just "salient concepts neuron" stuff?
Like if you activate some concept strongly, and then ask "who are you," coherent responses will often sound like "I am X," where X is the concept. Right?
Am I missing something here?
Some thoughts on reviewing: spylab.ai/blog/reviewing/
First for @satml_conf where we trialed an early reject phase which cut submissions by ~40%.
Then more generally about how not everything deserves peer-review, and why we should reward writing things that people actually read.
I never considered that this might be possible even with training, but in hindsight it makes sense. The idea that it could in principle happen in the wild without being trained for is wild to think about
What a fantastic research idea! You can train a model to detect tiny numerical variations that arise from adding read-only interventions to the forward pass (which change the computation graph).