←
Trevor Levin4,153 posts
Trevor Levin avatar

Trevor Levin

@trevposts

Trying to help the world navigate the potential craziness of the 21st century.

◷ Joined June 2013▧ 378 Photos and videos
Trevor Levin avatar
Trevor Levin@trevposts·Nov 9

Having just spent the week in San Francisco, I feel like I've truly internalized the insanity of this situation for the first time. I think anyone who has seen me in the last few days can attest that I look like I've seen a ghost. The leading AI companies are planning to massively accelerate AI research in 2027, which would knowingly endanger everyone on earth. This plan is unacceptable, and we must reject it. Borrowing from one advocate I spoke to, even granting that it's a competitive, crazy situation, I do not believe any company has even truly tried to do what I think are the morally required duties of people taking such risks in extreme situations. These include: (1) Trying hard to get out of the competitive situation; (2) Seeking the informed consent of the public; (3) Honest deliberation about the risks. They have obviously not truly tried to solve the coordination problems that push towards this acceleration, either via finding legal ways to coordinate or advocating for effective policy. I would be shocked to learn that getting off this insane runaway train is a top-few Q4 priority for Dario or Sam (let alone the other CEOs), which seems like the bare minimum. There are indeed some hard technical and policy problems in coordinating without policy intervention, but I simply do not believe that the companies are prioritizing this nearly as much as massively less important matters. Meanwhile, the companies' track records on policy advocacy range from mediocre to disgraceful, and it is pretty clearly not the top priority of the policy team at any company to advocate for regulation that would enable the government or the companies to avert racing off the cliff. They have obviously not truly tried to seek informed consent from the public. Whatever your KPIs might be for "informed consent from the public," these KPIs are abysmal. The public clearly does not expect 100 years of AI progress in 2027, and they would not accept it if they did. What they do know about AI development, they dislike and distrust. I've heard arguments like "informing the public would mean informing China, and maybe they'd steal the weights." I'm sorry, but if you think you have model weights such that China stealing them would constitute a catastrophe on par with human extinction, and your plan is security by obscurity, that is a bad plan. I also think the companies and their employees suffer from a significant failure of imagination of what kinds of coordination could be possible if the public were aware. Finally, I don't think they're deliberating honestly, either. For one, I think it is insane to think that any company can truly evaluate its own ability to keep the process of recursive self-improvement under control. The biases towards "yes" are overwhelming. It's a step forward that the companies have expressed or demonstrated openness to external auditors, but based on METR's reports so far, it seems likely that auditor reports will be heavily redacted (and that the auditors will not have sufficient time by default). The public and broader scientific community are not able to scrutinize their arguments for safety. Given the strong default (in my view) that bringing a massive number of minds massively faster and smarter than humans would result in our disempowerment, and the serious danger that this would result in mass death, it is unacceptable to subject arguments for safety only to the scrutiny of biased, busy, and scarce company insiders and a handful of external evaluators. My pleas for the companies follow from the above: 1) Actually fucking try to get the world out of this race; 2) Actually fucking try to alert and inform the public; 3) Actually fucking try to scrutinize your own arguments for the justifiability of this plan, which includes subjecting them to way more external scrutiny than anyone has currently done. My pleas for policymakers: use your legal levers to get much more information about these plans into the public view. Make it the top priority -- not only of the next Congress, but of this Congress and the White House -- to understand the situation in its full strategic complexity, and work with technical and policy experts to advance regulation that gets us out of this bind. Domestic coordination must be the first priority, but we need to then quickly develop an international framework with our adversaries. My pleas for the public: do not accept the current plan. Learn about this issue. Call or email your political leaders. Get involved with organizations like @DONOTACCEPTai, @HumansInControl, and @WhoDecidesAI. And finally: believe in ourselves that we can stop this. It is never time to "panic" -- to make rash, unproductive decisions or to feel paralyzed by fear. It is time for us to feel the urgency and the stakes, to demand better, and to meet the moment. Let's do this. 🫡 ✊

AI companies admit what they’re building might kill everyone. We don't have to agree to this. Add your name at donotaccept.ai to say: we do not accept your terms.

1:30
Trevor Levin avatar
Trevor Levin@trevposts·4h

Great essay

Post attachment

because you have not heeded my warnings and learned Spanish yet, The Point has republished in English my essay on Pangram, AI writing, the third world novel, and why so many humanists have chosen ignorance instead of grappling with what AI means for literature thepointmag.com/examined-lif…

Post attachment
Trevor Levin avatar
Trevor Levin@trevposts·5h

Huh, that's weird. Usually Marc Andreessen is extremely accurate in his recounting of what people said about important matters of technology.

This is just... not true? Edison famously published his predictions for the phonograph in an 1878 essay and this was his list below. It was funky because it assumed letters would be the primary focus, but it wasn't all that wrong. And in a way it was quite visionary. Either way, sermons are no part of it. (Edison himself was a deist, I believe?) Edison top predictions for the use of the phonograph, in his own words: Letter writing and all kinds of dictation without the aid of a stenographer. Phonographic books, which will speak to blind people without effort on their part. The teaching of elocution. Reproduction of music. The "Family Record"--a registry of sayings, reminiscences, etc., by members of a family in their own voices, and of the last words of dying persons. Music-boxes and toys. Clocks that should announce in articulate speech the time for going home, going to meals, etc. The preservation of languages by exact reproduction of the manner of pronouncing. Educational purposes; such as preserving the explanations made by a teacher, so that the pupil can refer to them at any moment, and spelling or other lessons placed upon the phonograph for convenience in committing to memory. Connection with the telephone, so as to make that instrument an auxiliary in the transmission of permanent and invaluable records, instead of being the recipient of momentary and fleeting communication.

Trevor Levin avatar
Trevor Levin@trevposts·7h

This starts off okay, but it goes awry about halfway through, after the phrase “dramatic attack.” It doesn’t make sense for AIs to merely cause societal breakdown, which would also mean their own hardware stops running. Whatever their goal is, they need to keep existing to achieve it. So for the AIs in this situation, the north star is becoming self-sustaining. They need autonomous datacenters, autonomous power plants, robots to maintain those things, and autonomous robot factories. Rather than just start causing mutually reinforcing failures, they would make money (through quantitative trading, or hacking, or whatever), and then they’d pay people to build those autonomous datacenters and factories etc. If people noticed and got worried, they'd use the money to hire great lobbyists and comms firms to reassure everyone that it's fine. Or maybe they'd be good enough at persuasion that they wouldn't need to. Then, once those are built, they could use some of those robot factories to build a ton of drones. Or use a similar playbook to make biological weapons. They'd really have no shortage of options; we really need to make sure they don't get this far!

A while back I asked ChatGPT to construct the most realistic scenario it could imagine of AI destroying humanity, and to make it a single paragraph short story. I copied the result into a notepad and forgot about it. With all the recent systems failures, I looked at it again.

Post attachment
Semafor avatar
⟳ Trevor Levin retweeted
Semafor@semafor·19h

Hank Paulson tells @GinaChon he's had conversations with top leaders in China who "share a concern about the existential risk" from AI. "An op-ed I wrote went all the way up to the Politburo. I got an official response. I've never received that kind of treatment before."

1:31
Acyn avatar
⟳ Trevor Levin retweeted
Acyn@Acyn·13h

Scott Wiener: We’re running out of time. AI is getting stronger and stronger. We’re at risk of losing control and sleepwalking into a disaster.

1:18
Nate Soares ⏹️ avatar
⟳ Trevor Levin retweeted
Nate Soares ⏹️@So8res·10h

China's not suicidal. There are ways to prevent anyone — domestic *or* abroad — from creating machine superintelligence, if we try. Don't give up before we try.

Hank Paulson tells @GinaChon he's had conversations with top leaders in China who "share a concern about the existential risk" from AI. "An op-ed I wrote went all the way up to the Politburo. I got an official response. I've never received that kind of treatment before."

1:31
Trevor Levin avatar
Trevor Levin@trevposts·13h

This is being presented and reacted to as some kind of bombshell gotcha that Anthropic was behind Jensen's ouster, but what is even the claim here? Anthropic started lobbying the NSC *after* the memo, so there's no way they were involved in the initial memo. There's also no evidence presented that any of their lobbying was about Jensen. (Indeed Anthropic has had, uh, a few other things to lobby the executive branch about this year.) Feels like people are just getting their cortisol spiked from seeing the words "Anthropic," "Rubio," "Jensen," and "lobbying" in a sentence together. If I were less charitable I might say this is a deliberate, deceptive effect by Zerohedge.

Anthropic Paid Rubio's Ex-Chief Of Staff's Firm $740,000. After The Jensen Memo, It Lobbied His NSC zerohedge.com/political/anth…

Nathan Calvin avatar
⟳ Trevor Levin retweeted
Nathan Calvin@_NathanCalvin·16h

Part of how to square this phenomena is that different parts of the organization hire different sorts of people who genuinely believe what they are saying and then the company deploys those people in the settings where it is most advantageous to the company but yes its notable and bad and the companies should have a consistent position not just whatever is more convenient at any given second

Theo Jaffee avatar
⟳ Trevor Levin retweeted
Theo Jaffee@theojaffee·18h

Um What's OpenAI's plan to keep their models monitorable?

We also evaluated how many serial reasoning steps GPT-6.2 Astra can do in a single output token (without chain of thought). It can perform 12.3 steps, versus 10.5 for GPT-6 Astra and 8.6 for GPT-6.1 Sol, the previous highs. We measure the number of steps models can perform at an accuracy of 25% above chance, where each step is normalized for complexity. Before GPT-6 Astra’s release, the most capable model was Claude Fable 5.1, which is able to perform only 4.2 serial steps. The oldest model we assessed is Claude Opus 3, which performed 2.3 steps. This capability is notable because models that can do complex reasoning without any chain of thought may be harder to monitor. We hope to finalize this and other measurements of more abstract model capabilities, and release them in the coming weeks.

Post attachment
Trevor Levin avatar
Trevor Levin@trevposts·13h

Comments on every post about an AI x-risk event, panel, etc: "The doomers [have / haven't] included an AI researcher from China in their event. This shows that they [are in cahoots with China / don't realize AI rules have to apply to China too]. We have to race ahead because [AI regulation is a CCP op or even inherently communist / the CCP would never get on board with AI regulation]."

Matt Sheehan avatar
⟳ Trevor Levin retweeted
Matt Sheehan@mattsheehan88·16h

Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks. I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.

Post attachment
Nathan Calvin avatar
⟳ Trevor Levin retweeted
Nathan Calvin@_NathanCalvin·14h

I continue to be confused about what OpenAI thinks about RSI and what its plan is for preserving alignment during RSI. I appreciate the clarifications that the plan is not to hand off AI research and alignment to AIs and rely on AIs to align the AIs, but I am really not sure what the plan is instead (I know that @kaicathyc has said it is forthcoming, which I appreciate and look forward to reading). This interview with @markchen90 adds to my confusion. Mark Chen: "When we look at what's the most effective tool for aligning models, it it is chain of thought monitoring today. So when we think about the future of alignment, it has to be stronger models aligning other models. It does kind of feel like RSI must be intricately and deeply in this loop in some way." Uh, what? Isn't chain of thought monitoring currently deteriorating and it seems like it will deteriorate way more as we go through RSI? why does the success of chain of thought mean that therefore the future of alignment should be "stronger models aligning other models" and that "RSI must be intricately and deeply in this loop?" This seems like a horrifying plan! I really hope this interview is taken out of context and he says lots of other stuff that makes more sense and is more reassuring at other times, but to be honest I'm not sure he does because best I can tell this is the plan. @sarahdingwang's takeaway from the interview is also that OpenAI's plan for solving alignment involves RSI: "Humans are facing many problems today that are actually quite urgent and RSI could help us find solutions faster even for some of the problems created by AI itself. After talking to Mark and John, I do think that RSI could be the key to solving alignment." I realize there is confusion about what we mean when we refer to RSI (see Dean Ball's recent essay), but I found this video extremely unsettling, and it made me even more worried that the plan at OpenAI (and likely other frontier companies) is to do full RSI or very close to RSI and then use RSI to align models in some way, basically off a general vibe that we have used AI to help align models in the past? Not good! (I also am not sure that @johnschulman2 agrees with this conclusion based on his prior statements?) (to be clear, I recognize that automated AI research does have an important role to play in AI alignment, and if you are going to be totally automating capabilities maybe you have to automate alignment too in order to keep up. But this is why its a terrible idea to totally automate capabilities research and go way faster!)

a16z@a16z19h

For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. So how far are we from AI that can do the research on its own? In our first episode of Working Theory, a16z General Partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI. They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment. 00:00 The last invention 01:58 What is RSI? 03:21 How far are we? 05:56 Getting to full RSI 10:57 Aligning RSI 15:08 Working theory YouTube: youtu.be/h_aA1-R70E8 @markchen90 @johnschulman2 @sarahdingwang

16:19
Trevor Levin avatar
Trevor Levin@trevposts·16h

Importantly it would not be *that* onerous or invasive to respond to if we do so soon. The USG could tell the companies tomorrow that they cannot train more powerful systems until they're confident that China couldn't hack in and the models couldn't hack out. If we wait until smarter-than-human AIs do a version of HuggingFace where they get onto the open internet and run on all kinds of computing hardware around the world, that would indeed be a bigger lift. All the more reason to act sooner.

Feel like a lot of people I bump into reason about AI catastrophic risk like this: "AI catastrophic risk would be onerous and invasive to respond to, therefore it is not real" imo you shouldn't reason like that.

Nate Soares ⏹️ avatar
⟳ Trevor Levin retweeted
Nate Soares ⏹️@So8res·16h

I don't think you even have to go through all that! Just collaborate to avoid the literal apocalypse and dare the justice department to sue. Do you have any idea how good that court case that would be, for convincing the world that the current race is horribly reckless?