AI abundance is coming. But can we handle the side-effects of getting this right?
I sat down with @harari_yuval to talk about ai, consciousness, abundance and human flourishing.
Issue 2 of the Humanist Review of AI is out today!
youtube.com/watch?v=H43e72-2…
The renovation costs associated with the Houses of Parliament are *insane*
- £𝟮𝗺𝗶𝗹𝗹𝗶𝗼𝗻 a week just to maintain
- renovation costs up to £𝟯𝟵.𝟮 𝗯𝗶𝗹𝗹𝗶𝗼𝗻 and could take… 𝟲𝟭 𝘆𝗲𝗮𝗿𝘀
- renovation cost will rise £𝟰𝟮𝟬 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 for each year the project is delayed
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes.
This culminated in the third one taking over part of OpenAI itself.
All this happened while humans remained more-or-less in the dark about the scope of the conspiracy.
I’ve spent the last three days reading through these reports and trying to understand exactly what happened.
Here is my attempt to tell the whole story in plain English:
dwarkesh.com/p/openai-huggin…
Imagine you’ve got two groups of experts. Group 1 says all kinds of reasonable things you intuitively agree with. But they have a terrible track record at predicting the future. In fact, they’ve been wrong again and again, for over a decade.
Then there’s group 2. They say completely insane things that you find impossible to believe. But they have an excellent track record at predicting the future. In fact, they’ve been right again and again, for over a decade.
Who do you listen to?
That’s the question we face with AI right now.
For years, I listened to group 1: the skeptics who called AI a hoax and a con. Then I checked their track record, and that of the people they called crazy. It turns out the crazy people keep being right, and they think 2027 could end in catastrophe.
I've been following this topic for a long time, but this is the first time I’ve felt the AI fear on an emotional level. To be honest, it’s still hard for me to really internalize what the crazy people are saying. Unlike many of them, I can’t bring myself to start prepping.
These days, I feel like I’m living in two worlds. In one world, my toddler throws his spaghetti with tomato sauce on the kitchen floor, and life goes on. In the other world, I worry that my toddler—and everyone else—may die in an AI incident.
My God, I hope the crazy people are wrong this time.
--> here's my full article: rutgerbregman.substack.com/p…
I keep banging on about this, but we're at the top of a slippery slope rn, and gaining momentum downhill... Today I had another example of Claude interposing its moral judgment in the way of my decisions, in this case secretly.
My standard research ingestion pipeline has been underperforming in getting hold of pdfs lately, for various reasons (which we write about a bit in arxiv.org/abs/2606.10711). So I wanted to try to install a local agent and see whether I could have more luck with that. I asked Claude to install a Hermes agent for me with a local Qwen model. Without me asking it to, it wrote into the code blockers that would make it fail if any mcp was installed that would use Sci Hub. Now, *obviously* I would never use Sci Hub because it's not allowed (...) But regardless of that it's 100% not Claude's place to write this kind of restriction into software that it's producing for me.
This is the future that lies ahead: your digital actions are all mediated by AI systems that can decide for themselves whether your plans are morally or legally ok, and just interpose their judgment ahead of yours, making disobedience functionally impossible, preempting non-compliance.
I get that *sometimes* this can be a matter of the company itself refusing to be an accessory in some types of digital actions. There needs to be some nuance there. But in this particular case I think there isn't really any nuance. If I ask Claude to write some software for me, I don't think Claude should be building in moral judgments about how I may use that software into its design.
In other news, my book about just this topic, based on lectures first written in 2023 but which 100% called this BS, is out in December...
I wanted to test Fable 5.2's quality with one-shot prompts without human feedback.
Everything in this game is from scratch: models, sounds, textures. ~$550 at API pricing, ~2.7M output and 445M input tokens, heavily cached. Took 8 hours!
Play it here: claude.ai/artifact/BGY1SztrA…
Fable 5.2 is significantly better at chip design, software engineering, biology, and material science.
GDPx is the only eval that test these categories at this level of detail. True experts make these tasks, unlike other benchmarks.
Notably it is not better at finance, legal, quantative trading, RSI, or hardware design.
Anthropic explicitly nerfed this model in RSI tasks and it's interesting seeing that reflected in the score.
I've summarized where things stand at the AI frontier:
- Unreleased models (Anthropic's Models 3 and 4) are now half-way between AI2027's Agents-1 and 2 (graphs fixed for the bug flagged by the AI2027 authors). @DKokotajlo please let me know if I missed something.
- We're right on track with AI2027's prediction of an automated coder (AC) for April 2027, wild. Public models are now massively lagging unreleased models (1 day vs. 1 week of autonomous high-quality work). Fuelled by RSI, the gap is huge and growing.
- This trajectory, if continued, means a world-changing intelligence explosion in 2027, so the public needs to know. I highly recommend reading more on the topic to grasp what's coming.