←
Gabe Orlanski354 posts
Gabe Orlanski avatar✓

Gabe Orlanski ✓

@GOrlanski

PhD student @WisconsinCS / Intern @Snorkelai / RL Resident Prime Intellect

◷ Joined January 2021▧ 49 Photos and videos
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Sep 30

Agents will soon replace humans as the main users and designers of libraries. A good library will let future agents write correct programs with less code. Introducing LibraryDesignBench: one agent designs a library, and other agents write programs with it.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·17h

Thanks for having me! Was a great discussion on what the current state is and what the future holds!

Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

Our results with LibraryDesignBench show this gets WORSE with agent-designed libraries compared to those written by humans. Highlighting the need for closer design guidance and more focus on agents designing for agents.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

LibraryUseBench shows this is endemic when agents are given minimal guidance. Very rarely do they actively try to use libraries.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

More concerningly, we see agentic slop moving higher up the stack. ESPECIALLY around library usage, where some agents choose to reimplement entire libraries, even when they can install them.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

Of course, there are some new patterns emerging, but these have increasingly gotten more pedantic and subjective.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

The good news: sloppy patterns we observed a year ago are getting eliminated rapidly. Complexity buildup is slightly worse, but that is per model family.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

First, the benchmarks: SlopCodeBench tasks agents with building programs iteratively and penalizes poor design decisions. LibraryDesignBench requires agents to consider how future agents will use their library and design accordingly.

Post attachment
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Nov 12

A few weeks ago, @dexhorthy released a great blog on how frontier agents performed on SlopCodeBench. Now, slop is moving up the stack and destroying your codebase with duplicate code.

Post attachment
dex avatar
⟳ Gabe Orlanski retweeted
dex@dexhorthy·Nov 6

a year ago at @aidotengineer Code Summit I spoke about Research Plan Implement, how to apply context engineering to your coding agents, and why every concept in AI is doomed to suffer "semantic diffusion". That talk has over 600k views and changed the way engineers everywhere think about building with AI On thursday next week at 5pm, I'll be doing the ultimate mash-up talk: "Twelve Factor Factories: Design Patterns for Agentic Engineering" should be a fun one, see y'all there 🙂

Post attachment
Nicholas Roberts avatar
⟳ Gabe Orlanski retweeted
Nicholas Roberts@nick11roberts·Oct 30

How much of your training compute should go to rewriting your data as you scale pretraining? We trained a *22B* hero run at 33× our fit range and it ended up being within 1.1% of the pre-registered loss forecast! 🧵Thread from my amazing co-lead @AhmedSQRD 🧵 👇👇👇

0:13
Gabe Orlanski avatar
Gabe Orlanski@GOrlanski·Oct 7

Was thinking about this for a while today. And the NS moment for coding that come to mind for me are: Final METR time horizon update Rewriting Bun in rust Huggingface incident (ik not exactly coding)

vincent sunn chen avatar
⟳ Gabe Orlanski retweeted
vincent sunn chen@vincentsunnchen·Oct 7

Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment: - Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks - Launching the Open Benchmarks Red Team to continually test and strengthen them - Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development. Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases. We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders. We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…

Nicholas Roberts avatar
⟳ Gabe Orlanski retweeted
Nicholas Roberts@nick11roberts·Oct 6

I’m at #COLM2026! If you’re excited about the science of LLMs or scaling laws, feel free to reach out for a time to chat! Tomorrow at 4:30 PM, I’ll be presenting our work showing that joint pretraining and test-time scaling makes overtraining compute optimal. @COLM_conf 🦙

Post attachmentPost attachmentPost attachment