←
Brendan Halstead37 posts
Brendan Halstead avatar

Brendan Halstead

@bphalstead

Figuring out better plans @AI_Futures_ Co-author of AI 2040: Plan A and the AI Futures Model.

◷ Joined July 2026▧ 11 Photos and videos
Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

Other feedback loops, such as better AI capabilities increasing the supply of engineering labor, stack with this one, though the engineering labor channel will probably weaken as abundant labor saturates the experimental throughput achievable on limited compute.

Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

Hence, if the TasteVal doubling time applied to the situation where AIs are running the entire frontier lab, a "taste-only singularity" (or, an acceleration that's self sustaining solely on the basis of the research taste feedback loop) sole seems likely.

Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

The remaining factors in the left hand side depend on the historical rates of software progress, compute growth, labor growth, and spend shares/returns to scale in compute and labor. In the AI Futures Model with our median parameters, the value of those factors is 12.5, meaning that a 1/12.5 = 8% speedup to software progress per ECI point is required for self sustaining acceleration.

Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

It's somewhat unclear what the elasticity of software progress to this metric should be. In some sense, having 2x experimental taste in TasteVal means making 2x as much progress in the same amount of "time", using fixed experimental compute and coding labor, suggesting the elasticity should be 1. But here "time" only includes active GPU time, not wallclock time. The most conservative interpretation (which the authors use in section 5) is to interpret experimental taste as multiplying the number of GPUs you have available, to which software progress might have lower elasticity (like 0.6).

Post attachment
Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

What do the results imply about the strength of this feedback loop? The paper's "experimental taste" metric has recently doubled every 4.6 ECI points, so the elasticity of experimental taste to capabilities (in exp(ECI)) is ln(2)/4.6 = 0.151, or 15% per ECI point.

Post attachment
Brendan Halstead avatar
Brendan Halstead@bphalstead·Oct 6

The elasticity of software progress to researcher capability is maybe the most important quantity in RSI forecasting. IMO TasteVal provides the best evidence yet on the component of this quantity coming from *selecting better experiments* using fixed compute and coding labor.

Post attachment

How fast is AI's research taste improving? We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab. Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.

Post attachment
Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

Currently it's unfortunately unclear whether to take this seriously. [he might be conflating an acceleration in output [like AECI] with an acceleration in inputs [like lines of code?] Or it might just be confusing wording, like when he talks about "the accelerating exponential"?

Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

to the extent that "ai has been advancing drastically faster" in research *outputs* rather than research *inputs*, it seems valuable to publish a chart that demonstrates that.

Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

2. There could be an observed spike in pretraining software progress [none of which has actually made it into a finished model yet, hence no AECI acceleration]

Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

[but as of the risk report period ending july 15, their most capable internal model was worse than fable 5.1, so either that report excludes some super good "research prototypes", or this acceleration has happened primarily after july 15th.]

Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

There are some ways there could legitimately be a spike in research output despite the apparently nonaccelerating AECI trend. 1. There could just be a bunch of nonreleased models that are super good.

Brendan Halstead avatar
Brendan Halstead@bphalstead·Sep 12

Dario writes "since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI." But as of September 1, the AECI has not been "advancing drastically faster", so what does this mean?

Post attachment
Romeo Dean avatar
⟳ Brendan Halstead retweeted
Romeo Dean@romeovdean·Aug 5

One of my favorite proposals in this piece is: Limit capabilities of models used for AI R&D It seems relatively simple to enforce to me, and has some nice properties: > It delays the point at which AIs have the capability to sabotage AI research and align the next model to themselves. > It likely introduces a “negative internal-public gap,” meaning that models are deployed to the public before being used for internal AI R&D. This is good for societal response. > It has larger effects in worlds where takeoff is fast and therefore more risky. This also means that those who are skeptical of rapid recursive self-improvement may be open to this policy, because on their worldview it wouldn’t have as much of an effect. This policy could optionally be combined with a minimum transparent safety allocation

Post attachment

Plan A is an ambitious international plan for AI, but it's also good to consider medium-ambition options for domestic regulation. Especially after the Pacing the Frontier employee letter! So, we brainstormed a bunch of options and articulated them here: blog.aifutures.org/p/how-to-…

Brendan Halstead avatar
Brendan Halstead@bphalstead·Jul 29

Post attachment
OpenAI@OpenAIJul 28

At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone. We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible. pacingthefrontier.com