←
Charles Foster6,927 posts
Charles Foster avatar

Charles Foster

@CFGeek

On policy @METR_Evals 🧪 “excels at reasoning & tool use” 🪄 CoI disclosures on my Substack “About” page.

◷ Joined June 2020▧ 660 Photos and videos
Charles Foster avatar
Charles Foster@CFGeek·Sep 16

I often joke to people that I tweeted my way from “random AI researcher who likes open models” to a job at METR. But I don’t think I’ve told the story to y’all before. Time to do that: 🧵

Miles Brundage avatar
⟳ Charles Foster retweeted
Miles Brundage@Miles_Brundage·Nov 13

Very excited about AEF’s maturation and growth. It’s quickly becoming the default source of standards for third-party AI evaluation, not just in the US but globally — AEF-1 has been cited by the EU AI Office and Fangcun AI used the AEF-1 checklist in their DeepSeek assessment.

We are thrilled to welcome six new members to the AI Evaluator Forum! ➕ @ActiveSiteBio ➕ @ApolloResearch ➕ @CenDemTech ➕ @CommonSense's Youth AI Safety Insitute ➕ @GuidelightAI ➕ @SaferAI_org Each group brings scientific rigor and broadens the range of expertise at the table. Together, we will keep working to strengthen the field of independent AI evaluation.

Post attachment
Charles Foster avatar
Charles Foster@CFGeek·Nov 10

I like this framework

AI companies increasingly say they may soon reach recursive self-improvement (RSI). With RSI, AI capability progress could accelerate dramatically. This acceleration could happen inside AI companies before outsiders can notice. How close is RSI? Our new report reviews the evidence, turns key disagreements into forecasting questions, and gives initial LLM forecasts. 🧵

Post attachment
Charles Foster avatar
Charles Foster@CFGeek·Nov 10

h/t @S_OhEigeartaigh

This looks like a pretty good statement, and dovetails quite nicely with some of the problems we highlighted and recommendations we made last month. Good to see leading governments sign it, but also Anthropic, OpenAI and Mistral amongst others. Big emphasis on safety and security throughout the lifecycle of the model; information-sharing and other mechanisms for facilitating collective response; plus I was glad to see shutdown mechanisms and protected channels for whistleblowers to raise systemic risk concerns in there. It’s not everything that’s needed, but it’s something to build on. elysee.fr/en/emmanuel-macron…

Charles Foster avatar
Charles Foster@CFGeek·Nov 10

Another set of frontier AI safety commitments from companies and governments. IMO the notable bits are the push for a common framework (vs. company-specific ones) and fail-safe shutdown mechanisms that extend to training + internal use + external use. elysee.fr/en/emmanuel-macron…

Charles Foster avatar
Charles Foster@CFGeek·Nov 8

It’s easy to mistake linear gains in ECI for linear gains in model performance. Each additional point actually *multiplies* a model’s odds of success on every benchmark. For example, historically each 5-point increase in ECI corresponded to roughly a doubling in time horizon.

We’ve added ECI scores for Gemini 3 Pro, Opus 4.5, and GPT-5.2. ECI is computed from many benchmarks, but it correlates with other benchmarks too. So, we can use it to make predictions! Here’s what we get for METR’s Time Horizons. Gemini 3 Pro: 4.9h GPT-5.2: 3.5h Opus 4.5: 2.6h

Post attachment
Charles Foster avatar
Charles Foster@CFGeek·Nov 8

Like, this is great:

With better uncertainty estimates and a joint distribution over all models’ ECIs, we can now compare models directly. For example, we can now say Astra has a 72% probability of being more capable than Opus 5.5, but a 99% probability of being more capable than Fable 5.1.

Post attachment
Charles Foster avatar
Charles Foster@CFGeek·Nov 8

This is huge! Exciting

We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap. Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.

Post attachment
Charles Foster avatar
Charles Foster@CFGeek·Nov 7

The AI capability growth rate comes from a tug of war between “AI improves AI”, which pushes it up towards hyperbolic growth, and “diminishing returns” + “stepping on toes”, which pull it down towards a fizzle. Rate changes (modulo spending changes) give evidence of which wins.

[Claude helping me understand] Here's the same model written in terms of effective compute and research effort, with no logs. Quantities E, effective compute: hardware H times software efficiency S. Here "progress" just means E going up. R, research effort per unit time: researchers, human or AI, weighted by how fast and how good they are. 1. Research produces software efficiency, with diminishing returns dS/dt = a · R^λ · S^φ λ < 1 is the parallelization penalty. Doubling the number of researchers gives less than double the output. φ is the "ideas get harder to find" term. If φ < 1, each extra unit of S takes more effort than the last one did. 2. Automation ties research effort to effective compute With H fixed, AI labor grows with what that hardware can run: R ∝ E^α = (S·H)^α When S doubles, you can run twice as many copies, or the same copies twice as fast, so α ≈ 1. If smarter models also help, α is above 1. 3. Combine them dS/dt ∝ S^(λα + φ) Call the exponent n = λα + φ. Then: n < 1: growth slows down over time. Diminishing returns win. n = 1: steady exponential growth. S doubles every fixed interval. n > 1: faster-than-exponential, hyperbolic growth. S runs off to infinity in finite time. The condition n > 1 is the same as λ/(1−φ) > 1, which is Epoch's and Davidson's "r > 1" condition. What this means for doubling times The growth rate is (dS/dt)/S ∝ S^(n−1). Each time S doubles, the speed of progress gets multiplied by 2^(n−1). With n = 2, every doubling of S doubles the speed. That gives doubling times of 6 months, then 3 months, then 6 weeks, which is the figure/footnote's pattern. Those intervals add up to a finite total: 6 + 3 + 1.5 + … = 12 months. That's where the "within about a year" figure comes from. With n only slightly above 1, the doubling times still shrink, but slowly, and the blowup comes much later. The footnote's "rate exponential in cumulative progress" is the case n > 1 written in log units. In the units above, it just says the growth rate of effective compute rises as a power of effective compute. Before automation R was the human research workforce. It grows on its own schedule, independent of S. So dS/dt ∝ R^λ · S^φ with φ < 1 gives steady exponential growth that depends on hiring. Automation closes the loop by making R grow with S, which raises n. Where the real argument is φ: how quickly ideas get harder to find at the current frontier. If φ is quite negative, the loop fizzles. λ: whether a million parallel AI researchers duplicate each other's work and fight over the same compute for experiments. Being limited by experiment compute effectively lowers λ. α: whether extra S really turns into more research capacity. It does for inference, but not if experiments need training-scale compute, which H limits. METR's empirical claim amounts to saying n is currently above 1, or at least not clearly below it. The open question is whether φ falls as S climbs.

METR avatar
⟳ Charles Foster retweeted
METR@METR_Evals·Nov 5

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development. We are now publishing a report summarizing the results of this risk assessment.

Charles Foster avatar
⟳ Charles Foster retweeted
Charles Foster@CFGeek·Oct 30

New footnote with a bit more context on the timing of this publication!

Post attachment
Paul E Williams avatar
⟳ Charles Foster retweeted
Paul E Williams@PEWilliams_·Oct 27

I strongly encourage non-EAs who have great people skills and understand loss of control AI risks to apply to this job

I’m building a team in DC. Everyone in Washington has a stake in ensuring that reliable info about alignment and RSI reaches the public. We need trustworthy communicators who can explain METR's loss-of-control risk assessments and goals to policymakers.

Post attachment