←
Existential Risk Observatory ⏸1,629 posts
Existential Risk Observatory ⏸ avatar

Existential Risk Observatory ⏸

@XRobservatory

Reducing AI x-risk by informing the public. We propose a Conditional AI Safety Treaty: time.com/7171432/conditional…

◷ Joined March 2021▧ 170 Photos and videos
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

Many experts warn that AI can make humanity go extinct. But how would this happen? Our research, together with @MITFutureTech, @FLI_org, and fourteen co-authors, found eight scenario families. Today, we release our paper Existential Risk Scenarios from AI: A Systematic Review.

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·9h

It's been a true privilege to collaborate with the excellent researchers from @MIT_CSAIL! Review the eight AI existential risk scenarios we found here: aixrisks.com/

Researchers at MIT CSAIL & @XRobservatory have helped identify 8 ways AI could lead to existential harm. They mapped out 95 scenarios & grouped them into 8 scenario families, such as an AI takeover & automated warfare, potentially helping decision-makers prevent a global catastrophe: bit.ly/4kF44G7

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

Better scenarios may help objectify the existential risk debate, sharpen research agendas, improve policies, and help with resource allocation. If you don't know which problem you solve, your solution is unlikely to be optimal. We hope we contributed to better solutions.

Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

Finally, AI increases human destructive capability in catastrophic invention, where AI brings an existentially dangerous technology within reach, and automated warfare, where AI-driven war escalates beyond control.

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

AI can also enable permanent and global versions of existing risks: entrenched totalitarianism where an actor uses AI to secure its position, and economic stratification where AI enables permanent inequality.

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

In the coordinated takeover, AIs seize power by leveraging their positions and working together against us. Rather than a takeover, we may delegate authority to AIs ourselves. In this incremental handover scenario, taking power back from AIs passes out of reach gradually.

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

We found multiple AI takeover scenarios: in the unilateral takeover, power is seized through decisive capability advantage, possibly obtained by recursive self-improvement. In the dispersed takeover, individual systems seize influence after deployment.

Post attachment
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·20h

We reviewed 3,184 documents, identified 95 published AI existential risk scenarios and summarized these into eight scenario families. Each family is a sequence of causal steps. Browse the scenarios on our website: aixrisks.com/

Post attachment
Christiane Amanpour avatar
⟳ Existential Risk Observatory ⏸ retweeted
Christiane Amanpour@amanpour·Nov 13

The AI race “poses the greatest national security risk that our nation… but also the global order has faced in living memory,” says AI safety researcher @NPCollapse. “We are not in a race with China because we are fully compromised… Our data centers? Compromised. All of our models? Stolen.”

1:23
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·Nov 13

Building AI that can take over power and cause human extinction is obviously illegal under current law. Since public awareness, and likely judiciary awareness, of the topic is on the rise, it seems a matter of time until this can be enforced. Great work from @ir_replaceable_!

BREAKING: We’re suing @Anthropic, @OpenAI, @Google, @Meta + @SpaceX to demand an immediate freeze on advanced AI systems that threaten public safety. Together with @HeatInitiative, @Public_Citizen, and @secureainow, we’re going to court to stop these companies from building AI that could escape human control or enable bioweapons and cyberattacks. We won’t wait for a catastrophe to act. The people come before Big AI profits. apnews.com/article/artificia…

PauseAI UK avatar
⟳ Existential Risk Observatory ⏸ retweeted
PauseAI UK@PauseAI_UK·Nov 12

We are organising the largest AI safety march in history, and it’s happening right here in London. We want you to be there.

1:20
Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·Nov 12

We agree with @bshlgrs' warning below. A lot of people seem to be asking: but how will it take over power? Our @BartenOtto wrote his thoughts on the topic in @SCMP_Opinion last month: "ASI could spin up a vast amount of agents, creating AI armies where every soldier is superhumanly intelligent. It could run at superhuman speeds, making its relation to us perhaps comparable to our relation to plants. "Unleashed rogue ASI could make money, either legally by trading on the stock market or illegally by extortion or hacking entire banks or cryptocurrency platforms. It could also act in the physical world by using any of the internet-connected devices that we have, including weapons, or manipulating humans. "An ASI could plausibly hide itself on unguarded servers for a long time, overpowering us before we know it exists. Even more fundamental, a superintelligence might rapidly invent new science and technology. Once humans understood relativity theory, we were able to weaponise this understanding into atomic weapons. Which insights are still hidden to us but can be discovered and weaponised by a superintelligence, leaving us as exposed as a medieval castle to a nuclear bomb? "Besides AI strengths, there are also profound sources of human weakness. There are more than 12,000 nuclear warheads. Fragile human bodies are easy prey for chemical attacks or viruses, as we saw during Covid. We have even created companies which can synthesise these viruses and, absurdly, respond to email orders. Water or food could be poisoned, and hacked online communication networks could start blocking anti-AI coordination messages while sending messages that serve the AI’s interests while we don’t realise what’s happening. "If an AI needed us to maintain the electricity grid and server infrastructure, it could simply pay humans to do that, which many would accept because they need to pay the rent. Humans in powerful positions, for example in politics, could likewise be tempted to adopt pro-AI policies. "As if all of this is not enough, we are handing AIs huge advantages on a plate. When a new model is released, it gets used by customers including the military, government, financial industry, utility companies, media and many other powerful institutions. All that is then needed for a takeover is collusion by these AIs against us. "All these advantages and the fact humans are handing over total control to AI could well add up to the latter taking over power and, if it chooses, killing all humans." Full piece in the comments.

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim… I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo… If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

Existential Risk Observatory ⏸ avatar
Existential Risk Observatory ⏸@XRobservatory·Nov 10

Trump signals he may be willing to work with the Democrats to implement AI guardrails. What obviously needs to happen, and what will happen, is what we and many others have said for years: an AI safety treaty (or, if that's easier, coordinated national regulation) between the US and China, where public institutes vet models for risks we can't take both during and after development. This needs to be enforced via hardware regulation. One outcome of our research is that the hardest part of this is making sure we track and physically control all the chips that can train dangerous AI, globally. Regulators should waste no time making that happen. In addition, AI capabilities research is currently lowering the threshold of how many chips are needed for building such dangerous AI. It is crucial that this research stops, as it is undermining regulatability. The AI safety treaty will come anyway. But whether it is effective at saving us from extinction or other loss of control scenarios may well be determined by how soon and how effectively these measures are being taken.

Post attachment

Trump says AI risks can be managed, signals openness to guardrails reut.rs/4iMOeXn