.@coeff_giving directed over $1b in 2025, the most in our history.
I wrote a more personal annual letter this year, focusing on what a decade of grantmaking has taught me, how I'm thinking about AI, and how we're grappling with giving at this scale. 🧵
This is great, a game changer for medical tech regulatory reliance and efficiency in lower-income countries: a new “Global Reliance Disclosure Program” enables @FDA to “more efficiently share minimally redacted approval packages, including the FDA's detailed scientific and regulatory assessments, directly with companies who can then share them with regulatory authorities outside of the United States, empowering those authorities to rely on the FDA's rigorous, science-based review decisions rather than conducting duplicative and time-consuming product evaluations of their own, while maintaining sovereignty over their regulatory decision.” -
used for #lenacapavir but so much more potential! fda.gov/news-events/press-an…
Yesterday, the CCP's top theoretical journal (求是, Seeking Truth) published a major piece on AI governance and risks.
I think this is clearly the strongest & most urgent language on existential risks used in such an authoritative outlet. See below for comp to recent major doc.
Just read a Chinese article calling this "the darkest moment for Chinese AI." Seeing pushback in the comments, but basically it lays out the bear case quite extensively:
1. Getting compute is harder whichever way you go. Buying advanced NVIDIA chips is difficult, renting GPUs overseas is coming under pressure, and proposed restrictions on chipmaking equipment could make domestic alternatives harder to produce too.
2. Catching up could get a lot more expensive. If learning from leading models’ outputs aka distillaion becomes harder, Chinese labs have to spend more on their own data, experiments and training. That gets uncomfortable if investors are also losing patience. Independent labs don’t have an advertising or e-commerce business to pay the bills.
3. Robot-brain startups could find their advantage disappearing. Frontier labs are moving into robotics, potentially eating into what specialist models can offer. Meanwhile, robots still have to make economic sense against China’s relatively low labor costs.
4. More users don’t necessarily mean enough revenue. China has huge digital audiences, but getting people and businesses to pay sustainably for AI is another question. Everyone is competing on price, American models are getting cheaper too, and research isn’t getting cheaper.
5. Even the areas where China already leads face risks. Optical-component suppliers are benefiting from AI infrastructure spending, but proposed US restrictions could threaten access to major customers.
That’s a lot hitting at once. The worry is that companies keep making better models while needing more money to stay competitive, without earning enough to fund the next round.
Now, it feels like every six months it is a "darkest moment" for Chinese AI. But I think these are real concerns. We'll focus on this topic at @techbuzzchina (and present some hope, I think) next week.
@matt_levine wrote about AIUC’s plan for frontier lab risk to be governed by a mutual insurance structure.
A mutual solves anti-trust, letting labs legitimately collaborate specifically to reduce risks of incidents.
A great AI application that can make a big positive difference to people’s lives.
This Nature paper shows that an AI weather model can give people in the path of a hurricane one extra day of warning compared with the leading physics-based weather model.
nature.com/articles/s41586-0…
What does this mean for policy?
Many talk about cash transfers, but in most of our scenarios with divergence, they would have to be large to keep divergence from growing, 1-15% of advanced country GDP. Much larger than USAID was, or third wave AI philanthropy (0.1-0.3%)
GPT-6.2 Astra scores 174 on the Epoch Capabilities Index (ECI), on par with Fable 5.2 and joining it above what the pre-Astra trend for reasoning models would have predicted.
This is based on testing we did prior to launch using only our own benchmarks. The ECI is continually updated as more benchmark results become available, and so these values may change in the future.
My team at @AISecurityInst tested GPT 6.2 Astra before release, assessing its ability to evade monitors and solve problems without showing its reasoning. Read our findings in the system card: deploymentsafety.openai.com/…
Will economic growth from AI reach the developing world?
We think through possibilities and what they mean for policy and research: new paper with @LagakosDavid and @MitchVanVuren. 🧵
In The Argument, I write about a different big question about AI: what are we doing here? What are we hoping for? What would it even look like for this to go well? theargumentmag.com/p/what-ha…
The OpenAI Foundation is hiring an AI Verification Program Director!
AI verification is one of the most important fields of our time: verifiability will largely determine which agreements on AI development we can and cannot strike, especially internationally.
Apply here, or share with people who'd excel in this role: jobs.ashbyhq.com/openai-foun…
AI is driving a rapid increase in theoretical physics papers — submissions to arXiv are up 30% year over year. One physicists told @kagankans that the papers are "poorly written, but I think the science is probably pretty solid." understandingai.org/p/i-spok…
When I look at the LLM-generated forecasts on RSI indicators, I expect that, like most human forecasters, they are also dramatically underestimating AI progress.
I would personally give an 80% chance that we see a doubling in the rate of public AI progress by the end of 2027, and a 60% chance that there is a 4x acceleration relative to current base rates by the end of 2030 (conditional on no major policy response to slow AI after the first doubling).
(These are my personal forecasts and I do not speak for my colleagues. I expect my forecasts on these questions would be substantially higher than most AI experts’ forecasts, where I define 'experts' as we do in FRI’s Longitudinal Expert AI Panel.)
Whether AI is about to undergo recursive self-improvement is one of the most important questions in the world right now.
I was surprised that we couldn't easily find a living literature review on the topic that covers relevant evidence and forecasts, so we (@Research_FRI) put together an initial version.
We review evidence including:
• Frontier companies' claims about how much of their AI research is currently being automated by AI
• Public benchmarks that help us understand AI R&D-relevant capabilities
• Forecasts from prediction platforms, prominent researchers, and AI models
Overall, the likelihood of major acceleration in AI capabilities soon seems high. For example, AI systems give a 35-56% chance of the rate of AI progress at least doubling by 2030, and, after reviewing the evidence, I'd personally put the probability substantially higher than that (>80%). In our report, we share a framework for assessing whether a doubling in AI progress will eventually lead to broader RSI-like acceleration.
Some reflections from our research on this topic below. 🧵
AI companies increasingly say they may soon reach recursive self-improvement (RSI).
With RSI, AI capability progress could accelerate dramatically. This acceleration could happen inside AI companies before outsiders can notice.
How close is RSI? Our new report reviews the evidence, turns key disagreements into forecasting questions, and gives initial LLM forecasts. 🧵
Just read a Chinese article calling this "the darkest moment for Chinese AI." Seeing pushback in the comments, but basically it lays out the bear case quite extensively:
1. Getting compute is harder whichever way you go. Buying advanced NVIDIA chips is difficult, renting GPUs overseas is coming under pressure, and proposed restrictions on chipmaking equipment could make domestic alternatives harder to produce too.
2. Catching up could get a lot more expensive. If learning from leading models’ outputs aka distillaion becomes harder, Chinese labs have to spend more on their own data, experiments and training. That gets uncomfortable if investors are also losing patience. Independent labs don’t have an advertising or e-commerce business to pay the bills.
3. Robot-brain startups could find their advantage disappearing. Frontier labs are moving into robotics, potentially eating into what specialist models can offer. Meanwhile, robots still have to make economic sense against China’s relatively low labor costs.
4. More users don’t necessarily mean enough revenue. China has huge digital audiences, but getting people and businesses to pay sustainably for AI is another question. Everyone is competing on price, American models are getting cheaper too, and research isn’t getting cheaper.
5. Even the areas where China already leads face risks. Optical-component suppliers are benefiting from AI infrastructure spending, but proposed US restrictions could threaten access to major customers.
That’s a lot hitting at once. The worry is that companies keep making better models while needing more money to stay competitive, without earning enough to fund the next round.
Now, it feels like every six months it is a "darkest moment" for Chinese AI. But I think these are real concerns. We'll focus on this topic at @techbuzzchina (and present some hope, I think) next week.