←
Jon Bateman2,660 posts
Jon Bateman avatar

Jon Bateman

@JonKBateman

tech @CarnegieEndow / podcast The World Unpacked

◷ Joined February 2019▧ 179 Photos and videos
Jon Bateman avatar
Jon Bateman@JonKBateman·Nov 13

I talk to the great @FukuyamaFrancis about how liberalism can survive AI, Trump, China, and the Nietzschean “Last Man.”

As AI continues to progress, potentially indefinitely, does that mean we're coming to the end of the end of history? @FukuyamaFrancis and @JonKBateman explain, on The World Unpacked. Watch their full conversation here: youtube.com/watch?v=QUsJjUIf…

1:01
Jon Bateman avatar
Jon Bateman@JonKBateman·11h

tired: you can just do things wired: a thing is going to happen, you might as well be the one to do it

It's so crazy that Elon, who generally seems to think of himself as a Great Man of History, has indeed led companies that have pulled entire industries years forward and dismantled a federal agency in a week, and is the literal largest donor in American politics, has such a sense of helplessness about this. Call the president and explain that we're on track to soon have vastly superhuman AIs that we don't know how to control! We could ban RSI before Thanksgiving!

Matt Sheehan avatar
⟳ Jon Bateman retweeted
Matt Sheehan@mattsheehan88·17h

Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks. I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.

Post attachment
Jon Bateman avatar
Jon Bateman@JonKBateman·18h

Software is becoming less reliable and secure. It's also becoming more plentiful, flexible, and capable. I don't think "Software Beirut" captures this. More like Jack London's Yukon -- where some make quick fortunes, others die in the snow.

I haven’t lived here in awhile, but I see no other way to answer the hundreds of texts/DMs asking me if the world is, indeed, ending. In the span of six days, we’ve experienced; 11/12 — Microsoft outage 11/13 — Nationwide FAA ground stop 11/16 — Major US banking disruptions 11/17 — NSA’s classified internal network goes down We need far more analysis of why so much is breaking at once. I don’t want to get out over my skis here, and what I am hearing does not contradict the messaging that most of this is the result of screwy updates. But stepping back, the dots are starting to form a collective picture. We are seeing: • An explosion in severe and critical vulnerabilities, with AI accelerating discovery to 3X the number of vulnerabilities discovered in the same period last year. This is collectively being referred to as “vulnageddon” or “the vulnpocalypse.” • Mean time to exploit those vulnerabilities has collapsed to NEGATIVE SEVEN DAYS. AI allows attackers to find/exploit vulns far faster than software makers have time to fix them. • Powerful AI-enabled offensive tools like ARTEX have been thrown into the hands of every third tier nation state, cybercriminal and digital arsonist, which logically creates havoc. • This creates enormous pressure to compress patching windows and accelerate software updates, increasing the likelihood these fixes break things in unexpected, cascading ways. • Pressure to just “AI everything” means we are rolling out automated AI tooling on top of brittle infrastructure that can’t support it. My generation grew up in an extraordinary period of technological stability. We worked our way up the software stack, abstracting away layer after layer, until we lost a fundamental understanding of how the underlying systems actually work. We were told to “move fast and break things.” VCS told us to celebrate the idea that “software would eat the world.” Now we’re telling ourselves we can simply “roll out AI” across everything, as though nothing underneath will buckle. We never stopped to ask: SHOULD WE? Should we be digitizing every last nook and cranny of our companies, infrastructure, healthcare, homes, cars, bodies, lives? Especially when the complexity, interdependence, and fragility we were introducing overwhelmed our ability to understand/secure/maintain the systems we built? Nobody every asked: What if, in the name of progress, we were creating the ripest attack surface in human history? We are now entirely dependent on software systems that almost nobody understands end to end, that increasingly few people can repair, and that attackers can now probe at machine speed. We are hearing that incident response firms, the 911 for breach victims, are backlogged six months. Even those paid on retainer lack the bandwidth to triage every incident. The cavalry is disappearing. We are entering a period of unprecedented technological instability. “Software Beirut” is an incredibly apt analogy. We are in for a period of cascading failures, whether triggered by attackers, bad updates, or our own accumulated complexity. This, I’m sorry to say, is just the opening chapter.

Garrison Lovely avatar
⟳ Jon Bateman retweeted
Garrison Lovely@GarrisonLovely·Nov 16

Incredible news to wake up to: Obsolete is one of the Financial Times' best economics books of 2026! "What is astonishing, as this book makes clear, is how complacent we are about the multiplicity of risks that this creates." Writing a book with a strong perspective, there was always a risk Obsolete would be categorized as a polemic and discounted as such. But I aimed to make my argument with as much rigor and evidence as possible, and it's really nice to see it pay off! obsoletebook.org/ ft.com/content/45fcd5a0-54d2…

Post attachmentPost attachment
Jon Bateman avatar
Jon Bateman@JonKBateman·Nov 16

Commerce announces its own chip rule was "written by & at the direction of Treasury and NSC"... Never seen anything like this... Obvious question is, does the NSC represent Trump's own views here? Likely answer is, Trump not paying close attention ("Team of Vipers" phenomenon).

NEW: The Department of Commerce just published new AI and semiconductor export control regulations that close some major loopholes in existing export controls. The regulation includes new rules to: (1) counter Chinese smuggling of AI chips; (2) restrict Chinese companies from renting access to AI chips; and (3) block Chinese chip designers from making AI chips at TSMC. These rules are a major step forward. They still contain some key gaps that need to be filled urgently in a follow-up rule, namely, they do not block Chinese companies from buying AI chips inside the United States or renting access to AI chips at US-based data centers. However, this rule will help and is the first good-faith effort to actually close loopholes in export controls in a year and a half. Hopefully more will follow. 1. Countering Chinese smuggling of AI chips. The rule expands export controls on AI chips to five new countries: Singapore, Malaysia, Thailand, Indonesia, and Brazil. It does not block AI chip orders to these countries, but requires a US government review (unless it is a US company that meets certain requirements). This is a massively needed step. Singapore, Malaysia, Thailand, and Indonesia have been the second, fourth, fifth, and eighth biggest importers of AI chips this year. The rule doesn’t cover Taiwan itself, where chip smuggling is a massive issue (exports of AI chips to China are still not illegal under Taiwanese law), and still doesn’t prevent AI chip sales to Chinese companies in the United States, which somehow remain legal. Ultimately, large orders of AI chips by untrusted companies in any country should require a license to fully combat smuggling, as Chinese companies will move to buy AI chips anywhere there isn’t a license requirement. But this is a start. 2. Restricting Chinese companies from renting access to AI chips. The rule creates a new remote access control that restricts companies from providing Chinese companies with remote access to AI chips. It requires a US export license to provide Chinese companies with access to 12 million TPP or more (equivalent to ~200 of the most advanced Blackwell chips, or ~800 H100 chips). This applies to chips already shipped, as well as all future shipments. Some of these requirements even apply retroactively; Commerce says existing rules already barred data supplying a majority of a data center’s compute to Chinese companies, and tells companies that violated this to self-disclose. This effectively kills the business of neoclouds like Megaspeed that exist to provide remote access to large amounts of AI compute to China. The biggest gap is the remote access controls do not apply to data centers in the United States. So Chinese companies can legally train on large numbers of Blackwells at data centers in Northern Virginia. This gap should be closed immediately with a new rule that uses Commerce’s US persons authority to capture this activity. The rule also exempts recommendation algorithms, so impact will depend on enforcement. Last, new authorities included in the text of the AI Overwatch Act would strengthen Commerce's ability to enforce these controls--that bill should pass immediately. 3. Blocking Chinese chip designers from making AI chips at TSMC. The rule closes loopholes in existing controls that Chinese companies are likely exploiting to illicitly make AI chips at TSMC or Samsung. Specifically, it now requires any Chinese chip designer who is seeking to make a 16/14nm or more advanced chip at TSMC/Samsung to get a U.S. export license, and restores old rules that weren’t being enforced until now that require any untrusted company anywhere in the world to get a license to make an AI chip at TSMC or Samsung. This is a significant upgrade. Overall, this is a major step forward to countering Chinese smuggling of and access to advanced AI chips. The work here is not done, as there are some critical loopholes that remain open and need to be closed in a follow-up rule (namely those related to activities inside the United States). Commerce should move extremely quickly to close these. But this is the first time in over 18 months that the administration is starting to take export controls seriously, and does represent a meaningful step forward. Last, the fact that the Commerce press release says this rule was “written by and at the direction of the Department of Treasury and the National Security Council” is remarkable and unprecedented. I have never seen a BIS rule that was not written by BIS. That implies that Commerce was dragged kicking and screaming over the finish line by the White House and Treasury, and instructed to publish this rule rather than the rumored weaker one that it preferred. Remarkable.

Lauren Wilford avatar
⟳ Jon Bateman retweeted
Lauren Wilford@lauren_wilford·14 Sep 2025

people often post this stuff to imply that moral panics about the technologies of the past were quaint. But the past is full of reminders that we are on a long, slow march away from embodied experience, and that we’ve lost more of it than we can even remember

Daily Mirror, September 14, 1938

Post attachment
Jon Bateman avatar
Jon Bateman@JonKBateman·Nov 13

Fascinating

⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met. We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.

Jon Bateman avatar
Jon Bateman@JonKBateman·Nov 10

You can haggle over the odds and timelines. Maybe it’s 10% chance over 25 years — who knows. The basic ideas are getting harder to dismiss. We’ve often heard about the climate crisis, the democracy crisis, and many others. If these are crises, then surely AI is at least that, if not a true emergency.

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim… I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo… If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

Andy Masley avatar
⟳ Jon Bateman retweeted
Andy Masley@AndyMasley·Nov 10

I basically feel pretty confident at this point that we're barreling toward something at least akin to the Industrial Revolution and a lot of public intellectuals are forming their opinions about it mostly via confused vague popular slogans that emerge like a game of telephone

Scott Singer avatar
⟳ Jon Bateman retweeted
Scott Singer@Scott_R_Singer·Nov 10

Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.

Post attachment
Mackenzie Arnold avatar
⟳ Jon Bateman retweeted
Mackenzie Arnold@MackenZ_arnold·Nov 10

It's worth remembering that some of the most valuable political philosophy has taken place outside the academy. It happened in pamphlets and manifestos and essays series in newspapers. And that's not by happenstance. One has to interact with the world and the shaping forces of the moment to see a path forward. When it comes to AI, many would-be political theorists will take themselves out of the arena by refusing to engage with the technology as it is. Others will be too quick to force old solutions onto a radically different problem. In a note of optimism: this problem has gotten less bad since the summer. So many more people are taking AI seriously. I hope to see the early fruits of that engagement in the next few months. x.com/MackenZ_arnold/status/…

Agree with this. It's hard to know for sure when you're in the middle of it, but this does feel like an age of revolutions—something akin to the late C18. To be alive at such a time, as a political philosopher, is an extraordinary privilege.

Rachel Glennerster avatar
⟳ Jon Bateman retweeted
Rachel Glennerster@rglenner·Nov 7

Democrats should not bring back USAID even though its closure was a huge tragedy. The world has changed dramatically since 1961 and we have the opportunity to build a better more modern foreign asst strategy that focuses on results, innovation, cost-effectiveness, and scale.🧵

My vision for diplomacy and USAID.

2:17