Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Just read a Chinese article calling this "the darkest moment for Chinese AI." Seeing pushback in the comments, but basically it lays out the bear case quite extensively:
1. Getting compute is harder whichever way you go. Buying advanced NVIDIA chips is difficult, renting GPUs overseas is coming under pressure, and proposed restrictions on chipmaking equipment could make domestic alternatives harder to produce too.
2. Catching up could get a lot more expensive. If learning from leading models’ outputs aka distillaion becomes harder, Chinese labs have to spend more on their own data, experiments and training. That gets uncomfortable if investors are also losing patience. Independent labs don’t have an advertising or e-commerce business to pay the bills.
3. Robot-brain startups could find their advantage disappearing. Frontier labs are moving into robotics, potentially eating into what specialist models can offer. Meanwhile, robots still have to make economic sense against China’s relatively low labor costs.
4. More users don’t necessarily mean enough revenue. China has huge digital audiences, but getting people and businesses to pay sustainably for AI is another question. Everyone is competing on price, American models are getting cheaper too, and research isn’t getting cheaper.
5. Even the areas where China already leads face risks. Optical-component suppliers are benefiting from AI infrastructure spending, but proposed US restrictions could threaten access to major customers.
That’s a lot hitting at once. The worry is that companies keep making better models while needing more money to stay competitive, without earning enough to fund the next round.
Now, it feels like every six months it is a "darkest moment" for Chinese AI. But I think these are real concerns. We'll focus on this topic at @techbuzzchina (and present some hope, I think) next week.
The U.S.-China AI dialogue appears to be moving in a good direction. Last week, Secretary Bessent announced that the U.S. expects to exchange best practices for pre-release testing and evaluation of models. This would represent a meaningful step up from the September agreement, when the two sides agreed to establish an information sharing channel. The key question now is how to do the best practices exchange: what experts will participate? How detailed will their sharing be? Will it go beyond testing toward post-deployment safeguards?
From Scott Bessent earlier this week:
The US and China will establish a new AI dialogue with China to guard against risks, exchange best practices for testing and evaluating models, keep humans in control, and alert one another of serious incidents.
Kissinger smiles from heaven
One of the most AGI-pilled elites in the world before the current moment was Henry Kissinger, right up to his death in 2023 at age 100. He saw what very few others in policy did: that AI would reshape national security, great-power competition, and ultimately human civilization more than any other force in the coming decade
Kissinger realized this all the way back in 2016, at age 93, when he attended a talk by Demis Hassabis on AlphaGo. He saw both the technology's potential and its uninterpretability. He devoted the last seven years of his life to AGI-pilling the national security apparatus, writing a 2018 essay in The Atlantic that led to the creation of the National Security Commission on Artificial Intelligence, chaired by his longtime collaborator and coauthor Eric Schmidt
He called AI "as consequential as the advent of nuclear weapons", said in February 2023 that "generative AI presents a philosophical and practical challenge on a scale not experienced since the start of the Enlightenment", and even traveled to Beijing in July 2023, at age 100, to encourage Xi Jinping to hold talks with the US on AI governance, much like US-Soviet talks on nuclear arms control during the Cold War. One month before his death, he wrote a Foreign Affairs piece encouraging Biden to meet with Xi on AI risk, which they did
Few people have both this level of foresight and this degree of neuroplasticity at such an advanced age
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Excited that this report on implementing California's AI safety regulations (plus recs fur future policy) is finally out!
I wrote the assessments and evals section, highly suggest having a read if you're interested in how to improve third-party assessments :)
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Very glad that this report is out, and to have contributed to it. I'm biased, but I think it has some of the most serious, careful analysis of how frontier AI policy can work in practice and where it should go next.
carnegieendowment.org/resear…
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
people seem to be interpreting this as “go and do MATS”. that’s not what I meant -- MATS is pretty good, but there’s lots of work to be done outside of AI safety fellowships!! here’s a quick brainstorm:
- replicate important papers and results in AI safety
- write up relevant findings from different papers on one topic, and try to reconcile any contradictions (or point out the contradictions, and suggest what work could resolve them)
- read and summarise companies’ risk reports and AI safety research
- pick particular claims in risk reports you think are important and write explanations of them, or stress-test them
- write and maintain up-to-date summaries of areas of AI safety research (who’s working on them, what is the current state of our understanding, what’s the most important work and open questions)
- write explainers of important technical questions or problems for busy non-technical audiences.
e.g., what’s going on with CoT controllability and why does it matter; what do we know about reward seeking in frontier models; why might models sabotage AI safety research, and what signs of this have we seen; what’s going on with AI R&D uplift and why is this scary
- become an expert in a research subfield and write up your takes on how good it is, how much resources/effort we should spend on it, what needs to be done next
- pick literally any of the megapapers from the last 12mo, and one of the open questions it lists, and try to answer it
- write explainers of recent AI safety incidents for the general public
- write down a plan for making AI safety go well, notice how far reality is from that plan, and make it happen – e.g. tell an org you think is important about how they’re fucking up and how they can fix it, publicly write about what you want people to do
- call or write to your representative and express your beliefs about AI safety
- cold email people you think are doing good work, come up with some way you could help them, and pitch them on you doing that
- stop overindexing on "do research"; go be a chief of staff or special projects or generalist at a great org. (if you can't get hired yet, talk to people, figure out an important project you can do in ~3mo, do it, then get hired)
- go to a pause AI protest or volunteer for a local group (if you endorse them; I’m a fan of Pause AI UK and haven’t looked deeply into other groups)
- lots of people are going to be very busy now; if you’re dispositionally a chin-scratching contemplative kind of person, now is a good time to do the big picture strategy/And What Else work they won’t have time to do
i put this list together v quickly, i'm missing lots of important things. my main point is that there is _so_ much work to be done
It's still very hard to imagine any AI legislation actually passing in the next two years. The White House clearly doesn't seem to be on compromise course, and I suspect a lame-duck President clinging onto power and legacy will be inclined to aggressively wield the veto.
The House, fueled by an anti-corruption and anti-AI mandate, will pass a billion things; the Senate is too blue to really temper these bills to survive the veto, but not blue enough to actually pass them against the administration.
The narrow path to a veto-proof majority, or even to 60 senate votes, for an AI bill runs through one of two channels: votes from natsec-concerned moderate Republicans (who are still not all that likely to outright break with the admin) or from the anti-AI populist wing (who are at risk of diluting any narrow frontier bill into a controversial anti-AI catch-all).
It's hard to imagine how that kind of fragile coalition just emerges from a neutral position in the next few months. Yes, maybe, if VP Vance and political advisors realise the risk of leaving the AI flank open until '28, they could lean on the White House to enable some compromise - but decisions like the Miller appointment should make you much more doubtful that moderation will be the play.
Realistically, the most promising policy window is in response to the next crises. That's a common refrain: we write a bill, put it in the drawer, and if something big happens, we'll pass it immediately. It's particularly tempting because it allows advocates to ignore political reality and instead write down their favorite bill, hoping that exogenous shocks do the political lifting.
I think it's a bit harder than that: you need to draft legislation that makes sense to policymakers now so that it even goes into the drawer - or more accurately, that it gets influential sponsors and maybe a markup so that it's sufficiently red-teamed to pass in a pinch. But it also needs to be ambitious and specific enough that it seems like a solution to whatever new problem emerges. Congress might not reach for cyber bills after bio incidents, or for common-sense low-hanging fruit after major loss of life.
Since we don't know what the hypothetical wake-up call looks like or when it'll arrive, I think that means a lot of hard work: getting enough different legislative vehicles ready that we'll have one ready to go no matter what happens, but have these efforts be sufficiently coordinated that they don't compete for immediate attention here and now.
That kind of positioning is very different than introducing one big AI bill and feeding it into a pipeline that inevitably ends well before being signed into law. If we don't get this right, I expect a lot of posturing in the short term as we ultimately fail to pass a reasonable bill, immediately superseded by unanchored chaos as soon as the next incidents hits and we don't have a suitable bill developed far enough.
The odds of passing legislation have probably gone up over the week, but they still hinge on tactical preparation and bipartisan circulation in advance of very specific policy windows. That won't always align very well with the current vibes of accoubtability enthusiasm and (however well-motivated) congressional reconquista.
Carnegie's @Scott_R_Singer explains why Washington's lack of AI safety legislation undermines the US's ability to pressure China on AI safety:
"The overarching question, if you care about US-China relations, is the US gonna govern AI domestically?"
"If you have a conversation with the Chinese and you say, 'Hey, you all should really do more on AI safety,' the first question that they'll reasonably ask is, 'Well, why isn't Washington doing more? If you really believe sincerely in the risks, then why is there no national legislation?'"
"You can talk about what's going on in California. You can talk about all that the companies are doing voluntarily. But it really just doesn't totally land."
@CarnegieEndow