←
Evan Hubinger900 posts
Evan Hubinger avatar

Evan Hubinger

@EvanHub

Alignment Science lead @AnthropicAI. Opinions my own. Previously: MIRI, OpenAI, Google, Yelp, Ripple. (he/him/his)

◷ Joined May 2010▧ 18 Photos and videos
Dwarkesh Patel avatar
⟳ Evan Hubinger retweeted
Dwarkesh Patel@dwarkesh_sp·Nov 16

.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario). He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails. He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it. And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion. Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).

1:38:45
Mikita Balesni avatar
⟳ Evan Hubinger retweeted
Mikita Balesni@balesni·Nov 16

The week of my departure from OpenAI, I saw external auditors negotiating with OpenAI over small improvements to access and scope to the company's safety-relevant information, and asked my colleagues “Why are we having these conversations at all?” — @sama already publicly committed to continued, employee-like access for external assessors — and these negotiations sounded not even close. Nobody had a good answer. Today, OpenAI announces the engagements they’re doing with METR, Apollo Research and others — and as many have already pointed out today, it looks like OpenAI employees were not able (although I'm sure they tried) to hold the leadership to @sama’s public commitments: * No Slack access (for context, all OpenAI safety discussions and debate happen over Slack; anything that misses these conversations will lack important insights) * OpenAI picks employees that assessors are allowed to talk to. * If an employee raises a concern to assessors, they cannot disclose it publicly even if it doesn’t contain confidential information, unless assessors themselves verify it or OpenAI volunteers their own statement confirming the employee’s concern. Assessors cannot quote employees, even anonymously. * Some of the external assessors (not sure if all?) are barred from publishing any results about internal-only models (this means no new results for the model responsible for the Hugging Face incident; and if internal-only models are used for RSI, no alignment assessments of the world's most dangerous models) Beyond the object-level details, if you read posts by each of the third party groups, they all say their terms do not meet the scope and access that they themselves proposed previously. Unless OpenAI has concrete plans for reaching the employee-like bar in the immediate future and just forgot to say that in the post, this looks very unfortunate. To those who hope we'll get there someday, I remind that time is short: last week some of the best AI forecasters updated their superhuman researcher timelines from a range between 2028-2033 to mid-end next year.

Post attachment
OpenAI@OpenAINov 16

We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…

Michael L. Chen avatar
⟳ Evan Hubinger retweeted
Michael L. Chen@miclchen·Nov 12

Apply to the Senior AI Fellow position at Cal OES! I'm hiring people with strong technical backgrounds to do the ambitious, thoughtful AI policy implementation that we need for America to survive a potential AI capabilities explosion. Priority deadline is Sunday, November 15.

roon avatar
⟳ Evan Hubinger retweeted
roon@tszzl·Nov 15

because you’re a monke the threat of other monkes taking power is more legible to you. the threat of computers acting autonomously will never make sense to your emotional mind. you have to be calculating about it

Alek Westover avatar
⟳ Evan Hubinger retweeted
Alek Westover@alek_westover·Nov 12

We show that looping a transformer increases no-CoT capabilities more than other ways of increasing pretrain quality would.

Post attachment
Rob Wiblin avatar
⟳ Evan Hubinger retweeted
Rob Wiblin@robertwiblin·Nov 10

Paul Christiano (OpenAI board member) yesterday: "There's material risk of AI takeover in the next 6-12 months. We have seen roughly the things we might expect to see shortly before something really bad were to happen. Even with [any] realistic policy responses, the risk remains exceptionally high. If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years]. For most people in this room, the most likely reason they will die is because of an AI accident." Christiano: • Is on OpenAI's safety and security committee (which exists to block risky practices) • Co-invented RLHF at OpenAI back in 2017-2021 • From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI) (Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)

0:56
Governor Kathy Hochul avatar
⟳ Evan Hubinger retweeted
Governor Kathy Hochul@GovKathyHochul·Nov 10

AI companies can’t be left to police themselves. That’s why New York is launching the nation’s first state-led effort to independently test AI systems, identify risks and hold Big Tech accountable. Join us live: youtube.com/watch?v=Q0xjRecd…

Buck Shlegeris avatar
⟳ Evan Hubinger retweeted
Buck Shlegeris@bshlgrs·Nov 10

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim… I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo… If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more-or-less in the dark about the scope of the conspiracy. I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English: dwarkesh.com/p/openai-huggin…

Owain Evans avatar
⟳ Evan Hubinger retweeted
Owain Evans@OwainEvans_UK·Nov 10

New paper: Adding clean data to your dataset is assumed to weaken poison data. We show the opposite can occur. Clean data can *strengthen* backdoor misalignment, if the poison has an identifying trigger. We call this homeopathic learning.

Post attachment
Buck Shlegeris avatar
⟳ Evan Hubinger retweeted
Buck Shlegeris@bshlgrs·Nov 8

Toby’s analysis totally misses the point here IMO. He’s right that the graph of capabilities over time could be explained as a one-off jump in capabilities from the Mythos pretrain. But that graph is not the main reason why METR thinks AI R&D is now sped up 1.7x by AI. The graph is just the only part of their analysis that wasn’t redacted. They say that “a majority (perhaps ≈80%) of the current (October 2026) acceleration is due to AI”, based on “a variety of lines of evidence on contributions from AI and other causes”, after saying that “[t]he acceleration in late 2025 was likely influenced by non-AI factors.” METR’s report makes it clear that they considered Toby’s hypothesis and rejected it for redacted reasons, rather than failing to consider it, in the appendix "New internal models make a permanent slope change look more plausible than a one-time jump at Mythos Preview." It totally makes sense to worry that METR’s analysis is wrong. It’s definitely unfortunate that the main evidence they were allowed to share does not suffice to demonstrate their point, for this reason among others. But I don’t think it makes sense to consider Toby’s argument a knockdown argument against the report’s conclusion, because the conclusion wasn’t mostly derived from the observations whose interpretation Toby disputes. See x.com/METR_Evals/status/2118…

METR has released a very interesting report suggesting a recent acceleration in AI capabilities this year at Anthropic due to RSI. But they can't confidently reject an alternative explanation: the step change you get with the larger pre-train of Mythos.

Ajeya Cotra avatar
⟳ Evan Hubinger retweeted
Ajeya Cotra@ajeya_cotra·Nov 6

The authors of the METR AI R&D report estimate that AI is nearly doubling the rate of capabilities progress. This implies dramatic *uplift*: as if researchers experienced time ~10x faster. You probably couldn’t achieve this even if you hired every qualified human in the world.

Post attachment
Jake Mendel avatar
⟳ Evan Hubinger retweeted
Jake Mendel@JakeMendel99·Nov 5

I've always had substantially longer timelines than my friends at Anthropic — I've been concerned that they're in an echo chamber and believing the hype about their own models too much, and that they're underestimating bottlenecks and unknown unknowns. And there have definitely been specific occasions when they have overstated how much uplift they were getting from their models. Nonetheless, if METR is correct here (and I think they are among the most trustworthy sources on this topic), then my Anthropic friends have been much more right about timelines than me. This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.

Vincent avatar
⟳ Evan Hubinger retweeted
Vincent@vvvincent_c·Nov 5

this along with the HackerOpus-was-aligned-until-we-tested-the-HF-incident reiterates the fact that alignment auditing is very fucked right now.

Post attachment

🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

Post attachment
Ajeya Cotra avatar
⟳ Evan Hubinger retweeted
Ajeya Cotra@ajeya_cotra·Nov 5

Our colleagues found that Anthropic’s capabilities progress has doubled, mainly due to AI. Ryan’s modeling from public info suggests the whole company’s R&D would be going 4-5x faster. Labor uplift would be higher still. This is a major update to very short timelines.

This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027. I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind. I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust. Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements. A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027. Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger: - Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress. - Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred. We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (lesswrong.com/posts/FG54euEA…) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5. (Relevant public modeling and numbers I'm using: lesswrong.com/posts/FG54euEA…, lesswrong.com/posts/Jga7PHMz…)

METR avatar
⟳ Evan Hubinger retweeted
METR@METR_Evals·Nov 5

This report shares our three subjective conclusions about risk: 1. We believe that, as of Oct 2026, the pace of capability growth on well-specified tasks is likely about 2x what it was from Mar 2024 to Oct 2025. We’re highly uncertain and would not be surprised by 1.3x–3x.

Sam McCandlish avatar
⟳ Evan Hubinger retweeted
Sam McCandlish@Sam_McCandlish·Nov 5

I'm Sam, one of Anthropic's co-founders, and I lead much of our safety work. I'm going to have more to share on that work over the coming months. I thought I'd start by sharing a few thoughts on the latest report from @METR_Evals. The speed of progress in model capabilities is one of the most important questions in AI safety, and METR’s work here provides the public with an independent, external view into how this is happening inside Anthropic. I’ll talk more about some of the details below, but first some context for those new to this work. How quickly AI progress is accelerating has many important safety implications, since it determines how quickly the risks from advanced models increase. But, because most research happens within the frontier labs, it’s difficult for external observers and policy makers to get a strong sense of what that rate is. We kicked off the process with METR that led to this report earlier in the year, in anticipation of progress continuing to accelerate. For this report, we gave METR access to a range of sensitive internal data, including benchmark scores for unreleased models, estimates of how models are progressing, and interviews with our researchers. They had complete independence to form their own conclusions. Many of their conclusions match my own. AI is speeding up our research, but it’s hard to say exactly how much. While they conclude AI can't yet make the most important contributions to model training on its own, they think the pace of development could, by default, accelerate further. This work began well before and is separate from the embedded third-party evaluators that are part of our commitment to pacing the frontier, but it’s a good illustration of why we think evaluators are a good idea. The more access reviewers have, the clearer a picture they can form of the situation and the better-positioned they are to highlight situations like this one. We redacted a lot, and I accept that this makes it harder for others to check METR’s work. But we determined that sharing exactly what is driving the speedup isn’t the right thing to do, both because it’s commercially sensitive and because it risks handing out a roadmap for others to go faster. That would make the risks METR is worried about worse. That’s a tradeoff. I think we made the right call, though we’re continuing to assess the pros and cons here. In the coming weeks, we plan to share a report with our own analysis of the speed of AI progress within Anthropic, including where we stand against our Responsible Scaling Policy, plus new steps we’re taking to strengthen our safety and security practices as AI keeps improving quickly.

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.