←
Joe Benton75 posts
Joe Benton avatar

Joe Benton

@JoeJBenton

Soon joining @METR_Evals | Previously Alignment at Anthropic, PhD at University of Oxford

◷ Joined February 2017▧ 5 Photos and videos
Joe Benton avatar
Joe Benton@JoeJBenton·Sep 11

I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going. Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards. I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development. I wrote up more thoughts here on my decision and what I hope changes: substack.com/@jbenton1/p-215…

Buck Shlegeris avatar
⟳ Joe Benton retweeted
Buck Shlegeris@bshlgrs·Nov 8

Toby’s analysis totally misses the point here IMO. He’s right that the graph of capabilities over time could be explained as a one-off jump in capabilities from the Mythos pretrain. But that graph is not the main reason why METR thinks AI R&D is now sped up 1.7x by AI. The graph is just the only part of their analysis that wasn’t redacted. They say that “a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI”, based on “a variety of lines of evidence on contributions from AI and other causes”, after saying that “[t]he acceleration in late 2025 was likely influenced by non-AI factors.” METR’s report makes it clear that they considered Toby’s hypothesis and rejected it for redacted reasons, rather than failing to consider it, in the appendix "New internal models make a permanent slope change look more plausible than a one-time jump at Mythos Preview." It totally makes sense to worry that METR’s analysis is wrong. It’s definitely unfortunate that the main evidence they were allowed to share does not suffice to demonstrate their point, for this reason among others. But I don’t think it makes sense to consider Toby’s argument a knockdown argument against the report’s conclusion, because the conclusion wasn’t mostly derived from the observations whose interpretation Toby disputes. See x.com/METR_Evals/status/2118…

METR has released a very interesting report suggesting a recent acceleration in AI capabilities this year at Anthropic due to RSI. But they can't confidently reject an alternative explanation: the step change you get with the larger pre-train of Mythos.

Joe Benton avatar
Joe Benton@JoeJBenton·Nov 6

Evidence about the rate of AI progress is an essential public good. Policymakers and society at large must know what's going on if we're going to respond adequately to superhuman AI systems in the next 1-2 years.

Joe Benton avatar
Joe Benton@JoeJBenton·Nov 6

That said, there’s still disagreement about how to interpret the data. E.g., Toby has a different conclusion. Unfortunately, given the report’s redactions we can’t adjudicate who’s right without deferring to the authors. (Imo deferring is reasonable, but you shouldn’t have to!) x.com/tobyordoxford/status/2…

METR has released a very interesting report suggesting a recent acceleration in AI capabilities this year at Anthropic due to RSI. But they can't confidently reject an alternative explanation: the step change you get with the larger pre-train of Mythos.

Joe Benton avatar
Joe Benton@JoeJBenton·Nov 6

Based on the report, many people now think it’s much more likely that AI companies will build superintelligence in 2027 or 2028. That would be a massive deal and it’s good the public knows!

This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027. I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind. I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust. Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements. A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027. Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger: - Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress. - Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred. We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (lesswrong.com/posts/FG54euEA…) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5. (Relevant public modeling and numbers I'm using: lesswrong.com/posts/FG54euEA…, lesswrong.com/posts/Jga7PHMz…)

Joe Benton avatar
Joe Benton@JoeJBenton·Nov 6

Having not yet started at METR, I had no idea this report was due to come out yesterday. I guess I got lucky with the timing. It’s good to see more public evidence about how AI development is accelerating.

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.

Joe Benton avatar
Joe Benton@JoeJBenton·Nov 4

Seems like it would be good for the public to have more reliable ways to know whether superintelligence is imminent than AI companies vaugeposting in their IPO prospectus.

We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.

Chris Painter avatar
⟳ Joe Benton retweeted
Chris Painter@ChrisPainterYup·Sep 16

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.

Josh Engels avatar
⟳ Joe Benton retweeted
Josh Engels@JoshAEngels·Sep 12

I left Google DeepMind's AGI safety team three weeks ago to join @METR_Evals. To some of my friends and family this seemed like a strange decision: I enjoyed the work I did at GDM and turned down offers from Anthropic and OpenAI. But I made the decision because of how high I think the stakes are right now. The AI companies are all trying to build superintelligence: systems vastly better than humans at everything. They plan to get there through recursive self-improvement, a process where AIs build even smarter AIs in a feedback loop. If this goes well, the resulting systems could be amazing at solving countless problems for humanity. But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic. And unfortunately, current AIs seem to be getting less aligned over time, not more. In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans. It’s not that these incidents were very dangerous in themselves. The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement. I now think that there's a terrifying chance that AI systems cause immense harm in the next five years. I don't know the exact probability, but I think it's high enough to make this the most important problem in the world. I think we need more time. That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are. That’s what I'll be working on at METR: studying where misalignment comes from in training, evaluating if current mitigations are sufficient, and investigating whether we’re on track to solve alignment at all. I think METR is doing exceptionally important work here, but it isn’t close to enough. I think it’s important that we have more organizations like METR keeping AI companies accountable and approaching these problems from different angles.

Joe Benton avatar
Joe Benton@JoeJBenton·Sep 9

This feels like a broadly accurate description of the situation: our industry may be on track to build systems that impose an unprecedented amount of risk on the world. Evaluating those risks, and changing incentive structures so we can address them, is why I'm joining METR.

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

Josh Engels avatar
⟳ Joe Benton retweeted
Josh Engels@JoshAEngels·Sep 8

Exciting update: I’m joining @METR_Evals to work on alignment incident investigations! My time at GDM has been amazing. But in light of recent incidents, I’m excited to build up public evidence for misalignment risk and get a better scientific understanding of model misbehavior.

Joe Benton avatar
Joe Benton@JoeJBenton·Sep 7

📄 As AI systems get increasingly powerful, it might be very important that they can help with key tasks like AI safety research. I think TASTE, this eval which @hasanbaig8 painstakingly built, may be one of the best signals of whether we're on track for our AIs to be helpful.

As we increasingly automate AI R&D, can we also automate AI Safety research? We built TASTE (The AI Safety Taste Evaluation), a benchmark to measure whether models can predict which research proposals experienced AI safety researchers prefer.

Post attachment
Joe Benton avatar
Joe Benton@JoeJBenton·Sep 2

I’m extremely excited about independent assessment of the risks from AI. It’s crazy that an AI company could currently undergo an intelligence explosion, or have a loss of control incident, without the public knowing. METR seems to have the best shot of changing that.

Joe Benton avatar
Joe Benton@JoeJBenton·Sep 2

An exciting personal update: Last week I left Anthropic to join @METR_Evals to work on embedded assessment of AI risks. Anthropic has been great to me. I’ve always known, though, that if something more impactful came up, I’d move on.

Joe Benton avatar
Joe Benton@JoeJBenton·Aug 13

The work that Emery, Caspar, Chi, and Alex have done to build this high-quality conceptual reasoning index is extremely impressive. Anyone working on scalable oversight, or eliciting models on skills that will be important during an intelligence explosion, should check it out!

We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?" Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this. Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality. This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below. Official leaderboard website which we'll keep up-to-date: conceptualreasoning.ai/

Post attachment
Mikhail Terekhov avatar
⟳ Joe Benton retweeted
Mikhail Terekhov@MiTerekhov·Jun 24

Our Anthropic Fellows project is now public! The labs are planning to hand off AI safety research to AIs, but can we trust these AIs? We explore a way to control them for "fuzzy" tasks like writing research proposals. This is a whole new direction in diffuse AI control!

Post attachment