Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-huggin…
I've seen a lot of compelling counterarguments to my RSI essay, and I want to write up a response to as many of them as I can manage.
But I'm sure I've missed a bunch of them.
If you've seen any interesting tweets/articles taking the other side, can you reply with their link?
.@paulfchristiano explains the basic AI takeover story:
Step 1. A rogue internal agent swarm gains effective control over OpenAI or Anthropic during the intelligence explosion
Step 2. By the point at which these AIs have exfiltrated themselves, and you've got billions of copies of super smart AIs that have compromised all web connected infrastructure globally, the government response (which these AIs are actively trying to undermine) becomes much harder to successfully carry out.
.@paulfchristiano on what the end output of the intelligence explosion (running within a datacenter) will be:
Billions of minds, thinking 10s of times faster than humans, learning from all their parallel experience, with better learning algorithms than the ones evolution gave us.
.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).
He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.
He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.
And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.
Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
People still underestimate how big of a deal fracking is.
After decades of America basing its geopolitics around foreign oil, fracking gave us energy independence practically overnight.
@DanielYergin explains:
Episode dropping tomorrow with @paulfchristiano.
He thinks there’s a 1 in 3 chance of extreme superintelligence within a year (absent any slowing down).
And he'll explain what labs and governments should do right now given that possibility.
Excellent essay. The appendix especially helped me grok, better than anything I’ve read, why AI R&D may actually be a verifiable task.
One analogy that came to mind as I was reading:
The usual arguments for why AI will be slow to cure diseases are about feedback loops. You need exposure to the real world, wet labs, slow, expensive clinical trials, etc., to know whether your hypothesis is correct.
Now imagine we had a full in silico model of the human body. Granular biomarkers on everything. Digital trials across millions of virtual patients, run 1000x faster than real time.
Almost all biologists would expect a mind-boggling speedup in medical discovery from this.
AI R&D ~already looks like that world. Dwarkesh’s appendix is basically a list of biomarkers:
- Pretraining → loss on held-out text
- Post-training → how quickly a model masters an environment it’s never seen
- Inference → GPU utilization, memory per token, latency
- Research taste → does a trick that works at 100M parameters still work at 10B?
As Dwarkesh puts it: “We can make models much better at things that can be tested inside a datacenter. And AI training is literally happening inside a datacenter.”
Once AI is as good as the best humans at AI research, algorithmic progress will get dramatically faster. There are three important reasons why: The automated researchers will be much faster, each of them will be very smart, and there’ll be a lot more of them.
Maybe to summarize the basic argument of this essay a different way:
Current AIs are spiky and jagged - worse than humans in most domain, but often as good, or better, in ones with verifiable proxies (for example, math).
One such verifiable domain (I argue) is AI research.
And while the first AIs that dramatically accelerate AI progress will be quite narrow (i.e. probably not able to automate most jobs), the consequences of that accelerated progress will be very general.
And that's because better learning algorithms are very general. For example, GPT-6 (while still being jagged) is just way, way better than GPT-3 at practically everything.
I'm arguing that we're only a couple years away (and possibly only a couple months away) from a process by which (initially) narrow AIs concentrate a jump as big as the one from GPT-3 to GPT-6 into a single year.
And if you got another jump that big, but starting from the base of GPT-7, I think you would end up with AIs that would rapidly become better than the best humans in every field.
Recently i've found myself getting more upset with discussions of AI risk on X
Like, often it feels like the sceptics are just dunking on ppl who are worried
And it's tough bc these issues are really complicated. I don't think there's any water tight case for an intelligence explosion happening soon, or it leading to AI takeover
So there are loads of genuine uncertainties and ambiguities, and of course sceptics can and should raise them.
But it's upsetting when they present them as slam-dunk proofs that the whole AI risk is flawed and silly.
Like, do you really think that no one has thought of this before? Are you actually curious about whether your objection holds up? Remember how most issues are really complicated, and ppl are often wrong about stuff?
And the bigger picture here is that hundreds of top experts inside and outside the industry think this will be a complete disaster. Like, that is obviously fucking scary and we should take it very seriously, not notice an apparent flaw in the argument and move on.
To be clear, AI safety ppl can be annoying too. Historically, I've found them much more annoying to be honest. I've felt pressure not to say stuff that vibes as 'anti-safety', and noticed ppl engaging in bad faith with me when i do.
I know none of this is specific to AI safety. Public discourse is dogshit on every topic. And probably X is waay better than average. But i still think it's a disgrace how low the standard is.
For a topic this important everyone on both sides should try to do better
Even superintelligence will need real-world experience to match top human experts. Reality has a surprising amount of detail, and you can't just think your way out of that.
But the AIs that come out of the intelligence explosion will be really good at learning from all their parallel experience
As a result, we should expect them to outclass expert humans within months of being deployed.
I call this the broadly deployed intelligence explosion.