* COVID THREAD ON BEST WE CAN DO LONG TERM *
Yes we’re in an acute crisis with Omicron and that needs dealing with, but it has actually made me want to think a lot about how this horrible pandemic ends, maybe you too? Let’s go through it…
1/35 (sorry, but this is troubling me)
I signed this statement for many reasons, but most importantly because I have three young children, and I do not accept any rationalization for knowingly endangering their lives or their futures.
I know the litany of excuses, but I do not see much evidence that Anthropic is trying very hard to coordinate a stop to the activity that their leaders believe is so dangerous.
Ahead of Anthropic’s IPO, a rare coalition across AI safety, open source, e/acc, science, and law has signed the MisAnthropic Statement. Read the statement or sign at: misanthropicstatement.com/
I signed this statement because I believe that Anthropic is in many ways the AI company doing the most harm. In public, their leaders warn of recursive self-improvement (RSI) creating uncontrollable systems that could cause human extinction within a decade. But in private, they are pedal to the metal on RSI, driving the race and boasting to prospective investors that it is their "competitive advantage". Ahead of today's IPO, prospective investors should consider whether the ROI is worth risking the lives of their loved ones.
Here are some ways in which Anthropic IMHO does more harm than competitors:
1) They’re driving and arguably leading the race to RSI.
2) With inflammatory rhetoric, they’ve done more than any other US company to pour gasoline on the geopolitical AI race, which harms US national security.
3) WaPo reporting suggests that they’re implicated in more civilian deaths than any other AI company (more than 100 little girls in a school selected for targeting by Claude as part of their Palantir collaboration). When Google left Project Maven after employee pressure, Anthropic chose to join.
4) They're the company leaning hardest into transhumanism and the idea that we should build conscious beings with rights, not controllable tools.
5) They and closely associated donors have done more than any other AI company to co-opt and neuter much of the AI safety ecosystem to back their stance: “RSI is very dangerous but let’s race to it anyway”. People have told me that they support today’s statement but are afraid of losing funding or other opportunities if they sign.
Our new MisAnthropic Statement is out, against @AnthropicAI using recursively self-improvement (RSI) to build Artificial Superintelligence (ASI), without humanity's consent or support.
If you're working for Anthropic, or investing in Anthropic, you're supporting one of the most dangerous and unethical companies in human history.
They're not even very good at their virtue-signaling and safety-washing. But they're crystal clear that what they're building could end our species, and they know it, and they're building it anyway.
‼️ @AnthropicAI is about to go public, and its IPO has united a noteworthy group -- including @huggingface co-founder @Thom_Wolf, @lessig, @beffjezos, @GaryMarcus, FLI leadership, & many others -- behind a new MisAnthropic Statement in response.
FLI co-founder @tegmark shared: "When it comes to recursive self-improvement (RSI), supposedly 'responsible' Anthropic is the most guilty of all the AI companies endangering everyone to enrich themselves... Ahead of this IPO, people should consider whether the ROI is worth risking the lives of their loved ones."
🔗 Read the full statement & sign at the link below ⬇️
We must immediately ban superintelligent AI and recursive self-improvement in AI, everywhere.
We’re letting frontier AI companies race to grow alien superintelligences we neither deeply understand nor reliably control.
If we don’t stop these reckless AI companies now, they may cause human extinction as soon as 2027.
Repost this if you agree we need a ban.
We need to stop the AI industry, including Anthropic, *right now*.
I'll be honest: for a long time I put my hope in Anthropic. It was founded by people who left OpenAI over safety concerns. And it's undeniable that Anthropic has done more safety research and advocacy than the other labs. Earlier this year I even praised its leaders for standing up to the Pentagon.
But I’ve started to wonder whether I was a fool.
In February, Anthropic cancelled the most important promise of its safety policy: to pause if its models outran its ability to keep them safe. The reason? The competition wouldn’t pause either. Anthropic keeps saying the industry must slow down, while it keeps racing faster and faster.
The company has committed nearly 900 billion dollars to more data centers. This month, Anthropic plans to go public at a valuation of more than 1 trillion dollars. Its own prospectus promises to “bring forth an age of miracles,” and warns in the same document that the technology may doom humanity.
But how do you pause a company that has to justify a trillion-dollar valuation to its shareholders?
There's no technical fix either. Evan Hubinger, the “alignment science” lead at Anthropic, said it publicly: the company has no plan for keeping the machine under control, and he puts the chance that AI kills everyone within a decade above 10 percent.
The things we’re seeing right now are exactly the things you’d expect to see shortly before something really, really bad happens. The leaders of Anthropic may think they’re playing some clever long game, but to anyone with a bit of common sense it’s clear that the net effect of all their actions is that they’ve massively sped up the AI race.
And if you're really so scared of the machine god, then here’s an idea you might want to look into: just don’t build the fucking machine god!
Read my full piece here: rutgerbregman.substack.com/p…
AI companies admit what they’re building might kill everyone. We don't have to agree to this.
Add your name at donotaccept.ai to say: we do not accept your terms.
The current Jensen Huang reaction reminds me of the Jacob Coxon reaction a few months ago --
Everyone paying attention pre-Coxon already knew that AI companies don't have a good plan for making superintelligence safe but are rushing ahead anyway, and that there's competitive pressure to keep going. Those people were glad that suddenly more people cared.
Likewise, everyone paying attention knew that NVIDIA captured Commerce long ago and that tons of GPU smuggling, loopholes, etc. were being ignored as a result. Again, now experts are thinking "finally something might be done"!
Something dramatic caught the public's attention, and the long-known issue became salient.
Here are two other AI-related things that everyone paying attention knows but most people don't, which may or may not eventually become salient to the public:
- There are no minimum safety and security standards in AI whatsoever. The law of the land is "publish your own policy and follow it." As one might expect, company policies are generally weak/non-committal as a result. Things like "don't train your model in a way that will predictably cause it to become hard to monitor (other than as part of safety research)" is not required, nor is monitoring AI agents' chains of thought for deceptive or malicious behavior, or taking serious measures to protect your IP from theft, etc.
- Since no federal AI whistleblower protection law has passed, there is legal ambiguity around whistleblowing in AI safety and security. If a company employee shares confidential information with a member of Congress about something that they consider dangerous, they may not be protected from retaliation if the information they share doesn't involve violating any law. Of course some people still share such things, but they are taking risks doing so, and Congress is systematically under-informed as a result, just as it ostensibly is gearing up to regulate this technology.
I’m still being a good boy wrt NDA. I think it’s reasonable to infer from this chart that for any level of intelligence, the compute required to attain it is declining superexponentially with time.
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.
It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.
Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.
It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
Imagine you’ve got two groups of experts. Group 1 says all kinds of reasonable things you intuitively agree with. But they have a terrible track record at predicting the future. In fact, they’ve been wrong again and again, for over a decade.
Then there’s group 2. They say completely insane things that you find impossible to believe. But they have an excellent track record at predicting the future. In fact, they’ve been right again and again, for over a decade.
Who do you listen to?
That’s the question we face with AI right now.
For years, I listened to group 1: the skeptics who called AI a hoax and a con. Then I checked their track record, and that of the people they called crazy. It turns out the crazy people keep being right, and they think 2027 could end in catastrophe.
I've been following this topic for a long time, but this is the first time I’ve felt the AI fear on an emotional level. To be honest, it’s still hard for me to really internalize what the crazy people are saying. Unlike many of them, I can’t bring myself to start prepping.
These days, I feel like I’m living in two worlds. In one world, my toddler throws his spaghetti with tomato sauce on the kitchen floor, and life goes on. In the other world, I worry that my toddler—and everyone else—may die in an AI incident.
My God, I hope the crazy people are wrong this time.
--> here's my full article: rutgerbregman.substack.com/p…
Paul Christiano (OpenAI board member) yesterday:
"There's material risk of AI takeover in the next 6-12 months.
We have seen roughly the things we might expect to see shortly before something really bad were to happen.
Even with [any] realistic policy responses, the risk remains exceptionally high.
If we build this as quickly as we can in the easiest way we can the chance is [more like] a coin flip [within the next 3 years].
For most people in this room, the most likely reason they will die is because of an AI accident."
Christiano:
• Is on OpenAI's safety and security committee (which exists to block risky practices)
• Co-invented RLHF at OpenAI back in 2017-2021
• From 2024-2026 was head of safety at the US gov's AI agency (now called CAISSI)
(Video is from 'The Hinton Lectures' there's another on tonight and tomorrow.)
By far the most alarming score on here is from a benchmark that doesn't get much attention. The Remote Labor Index gives freelance projects in a huge range of fields to humans and AI agents. When it was introduced a year ago, the best AI agent could only complete 2.5% of the projects. Opus 5.5 had the best score at 21%, and Fable 5.2 just doubled it, hitting 41%.
This is still a leading indicator of labor automation, as these projects are better defined than what you'd get at a typical white collar job. But the fact that one model generation can do twice as many projects end-to-end is a reminder of how fast change can come during exponential growth.
And Goldman Sachs is already reporting the first signs of labor automation outside of tech, based mostly on diffusion of models that are by now a few generations old.
When I looked into labor automation for my book Obsolete, there wasn't yet much evidence of widespread displacement. But the more important realization was that AI progress looks smooth on graphs but feels anything but, as capability tipping points are reached. Combustion engines improved for decades without affecting horses, but when they got slightly better, the horse population plummeted.
obsoletebook.org/
Fable 5.2 and Mythos 5.2 are a step above Opus 5.5 in almost all the capabilities we evaluated.
Testers have found that they communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Fable 5.1 and Mythos 5.1.
This is frightening and strongly indicates the time for redactions is over. @RyanGreenblatt if there’s really only 6-12 months left damn your contract and blow the whistle and share all redacted material you have.
In a new post, I walk through some very rough modeling (thanks to @RyanGreenblatt for lots of help and feedback!): planned-obsolescence.org/p/a…
I did not work on this project or see the unredacted report. All opinions are my own and do not represent METR or the report authors.