Agents will soon replace humans as the main users and designers of libraries.
A good library will let future agents write correct programs with less code.
Introducing LibraryDesignBench: one agent designs a library, and other agents write programs with it.
This work was supported by the @primeintellect RL Residency and by @snorkelai Open Benchmark Grants.
Thanks again to @dexhorthy for running SCB to get these results and to @omouamoua for the feedback.
Our results with LibraryDesignBench show this gets WORSE with agent-designed libraries compared to those written by humans. Highlighting the need for closer design guidance and more focus on agents designing for agents.
More concerningly, we see agentic slop moving higher up the stack.
ESPECIALLY around library usage, where some agents choose to reimplement entire libraries, even when they can install them.
The good news: sloppy patterns we observed a year ago are getting eliminated rapidly.
Complexity buildup is slightly worse, but that is per model family.
First, the benchmarks:
SlopCodeBench tasks agents with building programs iteratively and penalizes poor design decisions.
LibraryDesignBench requires agents to consider how future agents will use their library and design accordingly.
A few weeks ago, @dexhorthy released a great blog on how frontier agents performed on SlopCodeBench.
Now, slop is moving up the stack and destroying your codebase with duplicate code.
a year ago at @aidotengineer Code Summit I spoke about Research Plan Implement, how to apply context engineering to your coding agents, and why every concept in AI is doomed to suffer "semantic diffusion". That talk has over 600k views and changed the way engineers everywhere think about building with AI
On thursday next week at 5pm, I'll be doing the ultimate mash-up talk: "Twelve Factor Factories: Design Patterns for Agentic Engineering"
should be a fun one, see y'all there 🙂
How much of your training compute should go to rewriting your data as you scale pretraining?
We trained a *22B* hero run at 33× our fit range and it ended up being within 1.1% of the pre-registered loss forecast!
🧵Thread from my amazing co-lead @AhmedSQRD 🧵
👇👇👇
Was thinking about this for a while today. And the NS moment for coding that come to mind for me are:
Final METR time horizon update
Rewriting Bun in rust
Huggingface incident (ik not exactly coding)
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment:
- Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks
- Launching the Open Benchmarks Red Team to continually test and strengthen them
- Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods
Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development.
Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases.
We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders.
We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…
I’m at #COLM2026! If you’re excited about the science of LLMs or scaling laws, feel free to reach out for a time to chat!
Tomorrow at 4:30 PM, I’ll be presenting our work showing that joint pretraining and test-time scaling makes overtraining compute optimal. @COLM_conf 🦙