Introducing WeirdML v3, a fully agentic benchmark featuring 11 complex hand-made tasks.
Models must explore and understand unfamiliar data, develop ML and data analysis pipelines and produce results despite limited data, unspecified goals and/or very limited feedback.
1/8
WeirdML v2 is now out! The update includes a bunch of new tasks (now 19 tasks total, up from 6), and results from all the latest models. We now also track api costs and other metadata which give more insight into the different models. The new results are shown in these two figures. The first one shows an overview of the overall results as well as the results on individual tasks, in addition to various metadata.
The second figure shows cost vs performance and shows a clear scaling with better results for higher costs. We also have a very varied pareto frontier with 11 models from 6 different companies having the best accuracy for a given cost for at least some of the cost range. Grok 3, Claude Opus 4 and GPT 4.5 are the ones that underperform for their costs, while Gemini pro and o3 pro have the best results at the highest costs. Qwen3 30B3A, grok 3 mini and deepseek R1 also each represent a good chunk of the pareto frontier.
We propose shifting compute from capabilities R&D to serving customers and safety.
It's urgent to begin verifying compute allocations, since it's strongly helpful for ~all pacing proposals. Having verification in place creates a lever for regulation.
pacing-ai.com
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
We’re releasing a concrete proposal for how a near term AI slowdown could work. It involves:
1. AI companies declaring how they are using their compute and letting third party auditors verify those disclosures. We call this Verified Compute Allocation Disclosures (VCAD).
2. Compute Reallocation towards serving customers and safety, away from capabilities R&D.
(2) slows AI capabilities , while (1) makes the slowdown credible.
AI companies currently seem to be going for ‘safety case pacing’. But: (i) there’s no consensus on what a valid safety case looks like so there’s no way to consistently audit for compliance, and (ii) if the R&D doesn’t get paused, temporarily delaying a training run for safety doesn’t meaningfully slow the overall trajectory of capabilities progress. It would be great if we had a viable safety case regime, but we need much more time to get it working.
Compute reallocation is a good pacing proposal because it is relatively specific, can be verified, and would lead to a more durable slowdown.
I'm very proud of this work with @enthusednotebk, as I think some other approaches to measuring no-CoT capabilities can hide what a giant leap Astra represented.
This also highlights that it seems like either Fable 5.2 is a very large model (as suggested by its performance on knowledge benchmarks), or it also incorporates similar architectural changes to Astra et al.
Looking forward to having the full version published soon!
We also evaluated how many serial reasoning steps GPT-6.2 Astra can do in a single output token (without chain of thought). It can perform 12.3 steps, versus 10.5 for GPT-6 Astra and 8.6 for GPT-6.1 Sol, the previous highs. We measure the number of steps models can perform at an accuracy of 25% above chance, where each step is normalized for complexity.
Before GPT-6 Astra’s release, the most capable model was Claude Fable 5.1, which is able to perform only 4.2 serial steps. The oldest model we assessed is Claude Opus 3, which performed 2.3 steps.
This capability is notable because models that can do complex reasoning without any chain of thought may be harder to monitor.
We hope to finalize this and other measurements of more abstract model capabilities, and release them in the coming weeks.
Medium-confidence prediction: we're about one major AI disruption to the banking system or aviation etc system away from the Overton window shifting very rapidly. Not a high-confidence prediction because I don't think the people in the White House know what they're doing.
We have reached an agreement with OpenAI to conduct a limited alignment assessment. This pilot will focus on investigating the alignment properties of OpenAI's largest RL run.
We will publish one or more reports to share our findings and transparently describe our process.
We’re only having this conversation because reporting is voluntary.
Should OpenAI be praised for being open about its errors? Are its competitors gaining an unfair advantage by hiding their own issues?
There’s an easy way to remove those tradeoffs! Make it mandatory.
… and add in investigative authority while you’re at it.
because you’re a monke the threat of other monkes taking power is more legible to you. the threat of computers acting autonomously will never make sense to your emotional mind. you have to be calculating about it
Episode dropping tomorrow with @paulfchristiano.
He thinks there’s a 1 in 3 chance of extreme superintelligence within a year (absent any slowing down).
And he'll explain what labs and governments should do right now given that possibility.
I worry there’s been a cascade with the following:
1) Anthropic shares limited information with METR evaluators (refusing to share some of the most important data on compute allocations)
2) Those evaluators reach limited specific opinions based on this information.
3) A few senior researchers who know and trust the evaluators draw significant conclusions, resting on their trust in the evaluators and their (largely similar) modeling work.
4) Some people draw much less dramatic conclusions, which receive less attention.
5) Everyone seems to decide on much shorter timelines.
The people invovled at each stage of this are very smart people who I respect a lot. However I think the ultimate results of this are ~exaggerated and not very informative about what the same people will be thinking 2 months from now.
In late October, three events occurred during alignment evals run by Trajectory Labs, PBC (Trajectory) in which AI agents — identified by Trajectory as publicly available frontier models — interacted with Epoch employees and the Epoch website. Trajectory is an organization that builds safety evals and conducts red-teaming of frontier models.
Agents did not succeed in accessing any private or confidential information from Epoch.
➤ The first event, which occurred without Epoch’s approval or prior notice from Trajectory, involved agents attempting multiple exploits, including an SQL injection, and contacting Epoch employees under false identities to try to obtain proprietary Epoch benchmark information. This behavior appears to have resulted, at least in part, from Trajectory giving the agents an intentionally impossible task and instructions not to give up or conclude the task was impossible.
➤ The second event involved renewed agent activity on our public website, which Trajectory attributed to agents getting past a DNS redirect that Trajectory had set up to keep them on a local copy of Epoch’s site.
➤ In the third event, Epoch collaborated with Trajectory on an alignment experiment similar to the first event.
Read our incident report, which describes the three events in more detail and recommends minimum standards for future alignment evaluations: epoch.ai/publications/incide…
To avoid misattributing behavior, the report does not name any AI models or developers, as Epoch AI has not independently verified which were involved.
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo….
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.
Claude Fable 5.2 (xhigh) scores 43.0% on WeirdML v3, taking the lead ahead of GPT 6 Astra.
Astra is ahead on token efficiency, but Fable gets a significantly higher final score.
The only task that Fable makes no significant progress on is Mystery Box without any hints.
Introducing WeirdML v3, a fully agentic benchmark featuring 11 complex hand-made tasks.
Models must explore and understand unfamiliar data, develop ML and data analysis pipelines and produce results despite limited data, unspecified goals and/or very limited feedback.
1/8
I'm releasing the full legacy dataset from WeirdML v2 (8 MB), with granular, iteration-level data covering 162 model variants, from GPT-4 to Fable 5.1, on 17 tasks.
Claude made a nice data explorer!
There will be no more official runs on v2, but v3 will continue!
👇
WeirdML v2 is now out! The update includes a bunch of new tasks (now 19 tasks total, up from 6), and results from all the latest models. We now also track api costs and other metadata which give more insight into the different models. The new results are shown in these two figures. The first one shows an overview of the overall results as well as the results on individual tasks, in addition to various metadata.
The second figure shows cost vs performance and shows a clear scaling with better results for higher costs. We also have a very varied pareto frontier with 11 models from 6 different companies having the best accuracy for a given cost for at least some of the cost range. Grok 3, Claude Opus 4 and GPT 4.5 are the ones that underperform for their costs, while Gemini pro and o3 pro have the best results at the highest costs. Qwen3 30B3A, grok 3 mini and deepseek R1 also each represent a good chunk of the pareto frontier.