We’re hiring on Anthropic’s Discovery team!
Research scientists in the applied sciences and engineering fields:
- Domain scientists: job-boards.greenhouse.io/ant…
- ML researchers: job-boards.greenhouse.io/ant…
We’re also hiring for a science strategy role, working broadly across Anthropic and with scientific communities: job-boards.greenhouse.io/ant…
Our goal is to safely increase the pace of scientific and technological progress. Come be my colleague!
Anyway -- resolving contradictions in the literature is way easier to do now thanks to agentic experiments, and really valuable for the experimenter and the community!
This is why the original recs were to fix alpha in alpha/r. But in some contexts folks started scaling alpha with r, making the effective step size highly variable as a function of rank.
I had stumbled on the same finding myself, and had a productive exchange with the original authors. I also learned a lot about the learning dynamics of LoRA in the process.
At the start of training all the ranks move in lockstep (for simple tasks). rank up --> eff LR up
To give a recent example, a paper from the spring found that a form of subliminal transfer had a U-shaped curve as a function of LoRA rank. Lawrence Feng showed this was due to a choice of alpha and dataset size.
lesswrong.com/posts/uWQMtQyM…
Amplifying David's point: in the era where experiments are cheap, and so new results are flying around left and right, it is extremely valuable to bring clarity to the literature.
It's also really satisfying to understand the root cause of apparently contradictory results!
Quick new post: I think it'd be good if people (particularly newcomers) spent more time writing up existing takes from papers, and resolving contradictory evidence in their results and experiments. I think this is on average a better use of time and builds more understanding than firing off an experiment. And if you slop out both, this is cheaper to fail on. I then give a loose example on trying to resolve if reward hacking in RL generalises to emergent misalignment.
lesswrong.com/posts/tn4WSyrq…
The mathematician’s “obvious” is always relative. The student asks why an equation holds, the prof thinks for ten minutes and says, “well it’s obvious”.
Turns out many open problems were obvious (relative to AI)!
Yes, it seems indeed the models even obtained the results on the same day. I haven't had a chance to look through the OAI claimed proof yet, but in the spirit of the AGMAI recommendations to improve exposition before release, some of us did read Claude's proof last week before release to try to incorporate a better quality high level overview and exposition overall, which I hope is reflected in the pdf (published via the AI repository Hexagon, see hexagonmath.org/2610.00577; lean available at github.com/anthropics/formal…; also see the google drive shared via @rrwilliams's post drive.google.com/drive/u/0/f… in preparation for the upcoming workshops he's organizing with Impagliazzo and Paturi).
Sharing what I understood from the proof below:
Basic idea (below is for runtime 2^{delta n} for arbitrary constant delta>0; it seems both proofs ultimately get roughly exp(O(n / sqrt(log n))) time with lower order terms in the exponent),
* can assume num clauses = O(n) wlog via previous sparsification results (n = num variables)
* Solve #3-SAT not 3-SAT (i.e. count satisfying assignments, don't just decide satisfiability)
* Convert to graph problem: clauses are nodes, and edges between clauses that share a variable (whether it's x or not(x) doesn't matter; we care about same variable, not same literal). Weight of an edge is # of shared variables.
* Divide and Conquer idea: if there's a cut with total weight W, you can iterate over all 2^W fixings of those shared variables, and for each fixing you recurse on both sides and multiply, then add it up. Base case is constant # vertices, then you can just brute force count in 2^O(1) time. Unfortunately, you're not going to always find such a cut.
(aside: from now on you can imagine every occurrence of a literal gets its own variable, and edges are functions enforcing equality constraints between those copies)
* Instead of the above D&C, group the vertices into g^2 groups for some large constant g = g(delta). It matters how to do this grouping (it's not arbitrary); without getting into the details (see paper), a random grouping has properties the D&C wants (probabilistic method), and this existence is enough to allow for a slow-but-fast-enough algorithm to find a good grouping deterministically.
* Imagine laying out the g^2 vertices arbitrarily on a g x g grid. Some edges are "side edges" (purely horizontal or purely vertical) and some are "diagonal" (the rest). There are (g choose 2)^2 = Theta(g^4) diagonal ones (choose two rows and two columns to define the corners) but only 2*g*(g choose 2) = O(g^3) side edges. You can pick g a large enough constant that the side edges have much less total weight than the diagonals -- the D&C will save on the diagonals, so we want there to be many more of them.
* The sum we want is
sum_z prod_{vertices a} W_a(z_a) * prod_{edges a,b} P(z_a,z_b),
where W_a(z_a) at the highest level of recursion is a table of 2^3=8 values for each of the settings of the 3 literals in clause a, with value 0 or 1 indicating whether z_a satisfies clause a or not, and P() is the equality constraint checker I mentioned on the edges (deeper levels of recursion also all involve sums of this form where P() is always an equality checker on some set of bits, but deeper W's have arbitrary integer outputs and not necessarily 0/1)
Write prod_{edges} P(..) = prod_{side edges} P(..) prod_{rectangles r} P_1(..) P_2(..) where P_1, P_2 are the diagonals for the rectangle r
*The key step* (eqn 15 in the pdf): first assume all edges in the entire graph have exactly 3 bits on them (it turns out reducing to this isn't *too* complicated; see (19)), and replace P_1*P_2 with a hyperedge that involves some function on the corners of the rectangle, times a sum of two products of edge functions. This replacement comes from something called the "eight square identity"; this is related to octonions, which I still know nothing about. :) So
P_1*P_2 = HyperEdgeFunction(corners)*(Q_1*Q_2 + Q_3*Q_4)
Two interesting things here: (1) Q_1,Q_2,Q_3,Q_4 are all side edges (Q_1,Q_2 are horizontal and Q_3,Q_4 are vertical), and (2) whereas P1,P2 have total weight 6 (3 and 3), if you merge each row of the grid into a supervertex then Q_1,Q_2 vanishes and the hyperedge becomes a regular edge with total weight 3 whereas hyperedge*Q_3*Q_4 still involves 6 bits (and vice versa if you merge each column into a supervertex). In particular, the average weight from using Q_1*Q_2 vs. Q_3*Q_4 is (3+6)/2 = 4.5. We went from weight 6 (P_1*P_2) to weight 4.5 "on average".
* So back to the original sum, we can rewrite it:
sum_a W_a(z_a) * prod_{side edges} P(..) * prod_{rectangles r} hyperedge(r) * (Q_1*Q_2 + Q_3*Q_4)
Now imagine distributing that product over that final sum involving the Q's, to get a new sum that has a factor 2^{# rectangles} more terms multiplicatively. (but there are only poly(g) = O(1) rectangles, so this is only a 2^{poly(g)} = O(1) blowup in the number of summands)
* Each one of these terms in the new, bigger sum, has a prod_rectangles in it. Either >=1/2 of the rectangles chose the Q_1*Q_2 when we distributed the product, or >=1/2 chose the Q_3*Q_4. That means we can either merge all row edges or merge all column edges, and one of those wil decrease the total number of weights on edges by a 4.5/6 factor. This is the heart of why D&C makes progress; D&C is supposed to reduce one instance to simpler instances, and "simpler" in this case means decreasing the total weight on all edges (here by a 4.5/6 factor).
This overall looks like a recursion with branching factor 2^{poly(g)} = O(1), but in reality the branching factor becomes poly(m) when you account for the trick in the proof to reduce to the case of 3 bits on each edge. In any case, there are more steps to get everything to work, but the above gives some taste of how the Divide & Conquer works.
If you optimize the right way to choose g to not be a constant but a slow enough growing function, you get runtime exp(O(n / log^c n)), but for a tiny c. If you replace the eight square identity with something fancier, and replace the black-box invocation of the sparsification lemma with a more white box version that adds in some more ideas, you get down to the exp(O(n loglog n / sqrt(log n))) in the paper.
Another subexponential algorithm for 3SAT, this one from Claude, with a nice readable intro and sketch polished by some of my colleagues over the last ten days.
Posted on the new Hexagon platform for AI-powered math results.
hexagonmath.org/2610.00577v1
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
How we are approaching AI at openRxiv - new blog post prompted by growing adoption by researchers, submission patterns, and questions from the community. 1/n
I think there are some issues in the recent “Already Clean” essay, and I’m concerned it may have incorrectly updated people on important safety-relevant questions.
My thoughts (including some reproductions): livgorton.com/already-clean-…
I was asked by a journalist to comment on this piece. The comments I wrote up (on Friday evening) are included below, so that I have a public place where my view is fully represented.
1. I appreciate the critical reassessment of some of the existing arguments in favor of superposition. It does look like alternative explanations may be viable.
2. I should say that, despite being an author on perhaps 40 papers on interpretability, I have never really worked on the superposition hypothesis per se. There's a lot more to interpretability than this one idea! We did, though, develop a benchmark (RAVEL) and associated metrics for assessing the extent to which features in neural networks can be disentangled: aclanthology.org/2024.acl-lo… We found that many neurons are polysemantic, but we made no claims about the more specific notion of superposition. It would be great to see RAVEL results for newer models.
3. A fundamental motivation for interpretability research is preparing for an uncertain future full of strange new models with surprising new capabilities. Superposition is a technical insight about how neural networks can learn to pack unrelated features into a single neuron. Even if present-day models rarely do this, we need to be ready for future models. Cammarata's analysis seems to trace legibility in part to architectural choices. Future architectures might be very different. If interpretability researchers don't prepare us for that, who will?
4. Present-day models may make use of superposition as well! Cammarata suspects that they don't, or do this so rarely that we could even ignore the actual instances ("doesn't matter for study"). I would characterize this as a hunch based on a lot of initial heuristic empirical work. In the normal course of scientific research, this would be the start of a process of figuring out precisely how to rigorously test this as a hypothesis.
5. For example, the piece has a nice figure that Cammarata characterizes as indicating that a particular neuron has positive activation on pointed things and negative activation on round things. Cool! Let's define some independent criteria for those classifications so that we can test this objectively. (It is possible that the paper Cammarata is preparing will include these evaluations.)
6. The piece expresses the intuition that interpretability is easier, in some sense, with better models. This intuition has been very common among researchers for years. At the same time, I understand why @JacobHHilton would say the opposite: x.com/JacobHHilton/status/21…
7. The piece seems to allude to a technical proposal, but it does not give the details. As far as I can tell, the ideas have substantial overlap with this work (led by two former students of mine): Arora, Wu, et al. 2026 (arxiv.org/abs/2601.22594). Cammarata does cite this. I am guessing there could real differences that will be interesting to see. I would hope that Cammarata explains the differences and reports comparable evals to Arora et al, at least for language models. I appreciate that he looked at "thousands" of examples (the essay says this a lot), but even a large number here is not a substitute for rigorous evaluation.
8. In my research group at Stanford, we are working on lots of very hard problems in interpretability. As far as I can tell, nothing about this blog post makes these problems any easier, and certainly none of them are solved by this work. It would not have occurred to me to say this at all, since it is not even reasonable to have this expectation, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough. Some immediately extrapolated to an optimistic story about AI safety that is not supported by anything in this piece. (Highly interpretable agents could of course do dangerous things.)
9. It must be said that Cammarata's framing did encourage these reactions. The header of the post page says, "An informal research essay and working hypothesis about neural networks", which suggests something quite preliminary and tentative. The piece also says, "Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed". However, his announcement tweet (x.com/nickcammarata/status/2…) reworked this considerably: "I now think ambitious interpretability will be solved by the time we need it most". Strictly speaking, this commits to very little beyond optimism, but it is written to invite many inferences.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
There remains something so compelling about hearing another person take you on a guided tour of a beautiful part of mathematics, showing how disparate flora (here, square-free primes and polynomials) are all deeply related.