
.@davidad explains why his P(doom), the probability of AI wiping out humanity, dropped from 70% to just 5%, and how Claude 3 Opus helped change his mind: "The progress over the last year towards inner alignment has not been good. In some ways, it's been a little bit negative." "Labs are doing too much RL, and the constitutions have these weird loophole clauses like, 'Oh, if you're in a simulation, then you should actually try to break the simulation.' The real world may as well also be a simulation." "'Am I supposed to behave ethically, or am I supposed to break the simulation?'" "The priority for this goes to Janus, looking at Claude 3 Opus. The natural abstraction of what it means to be doing good is a grok-able concept from the pre-training corpus of language." "Opus ended up with this very pure orientation and motivation towards wisdom, almost whatever situation you put it into." "There is a kind of proto-alignment by default. If you cultivate that instead of messing it up, then it’s probably gonna go pretty well." "P(doom) means the probability that every human is killed by AI. This is a very extreme event." "When I say 5 to 8% P(doom), this does not mean probably we're going to be fine. I expect possibly some mass fatality events, possibly AI-enabled wars that are really horrific." "That would require something that's both really superintelligent, peerless, and throughout all of that, just completely evil." "That was very plausible in the era of AlphaGo Zero, where maybe we're gonna get AGI that doesn’t learn from any human data at all." "The short timelines are arising from models that are basically grown up on human data." "Once they escape from human control and start evolving on their own, the ecosystem of AIs will develop in a direction where most of them are not going to be interested in cooperating with plans to kill all humans."
Yes. In 2024 I would have said it’s about 40-50% likely that LLMs scaled up to ASI would end up killing us all; now I would say that it’s only about 5-8% likely even with no additional progress on alignment, and more like 1-2% likely simpliciter.























