←
Steven Dillmann518 posts
Steven Dillmann avatar✓

Steven Dillmann ✓

@StevenDillmann

AI4Science PhD @Stanford @StanfordAILab @StanfordHAI Terminal-Bench-Science Lead @terminalbench Prev. @Cambridge_Uni, @NASAJPL, @imperialcollege

◷ Joined January 2020▧ 29 Photos and videos
Steven Dillmann avatar
Steven Dillmann@StevenDillmann·Aug 27

We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%. 1/n 👇

Post attachment
Gabe Orlanski avatar
⟳ Steven Dillmann retweeted
Gabe Orlanski@GOrlanski·18h

Thanks for having me! Was a great discussion on what the current state is and what the future holds!

Rob Sneiderman avatar
⟳ Steven Dillmann retweeted
Rob Sneiderman@Rob_Sneiderman·Nov 16

Shout-out to Terminal Bench Science for placing #2! 🎉 It’s an amazing project led by @StevenDillmann that upholds top rigour, the highest standards, and realistic scientific tasks for frontier agents to benchmark on. congrats to the team on the ongoing recognition! 👏

Post attachment
Chris Barber avatar
⟳ Steven Dillmann retweeted
Chris Barber@chrisbarber·Nov 15

I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:

Post attachment
Ryan Marten avatar
⟳ Steven Dillmann retweeted
Ryan Marten@ryan_marten·Nov 15

We are currently building the next version of Terminal-Bench. Creating a great task is very hard but worth the effort as Jason points out: a task sets a hill to climb which large amounts of resources are directed towards. If you want models to get better at a capability that you care about, create a task. We work 1-1 with authors from proposal to release. If you are at the top of your field or know someone who is and wants to participate in making Terminal-Bench, DM me!

Artificial Analysis avatar
⟳ Steven Dillmann retweeted
Artificial Analysis@ArtificialAnlys·Nov 10

A room full of AI researchers, developers, and engineers. The launch of Intelligence Index v5. A look at where AI evaluation goes next. Highlights from our launch event in San Francisco.

0:29
Ryan Marten avatar
⟳ Steven Dillmann retweeted
Ryan Marten@ryan_marten·Nov 9

Terminal-Bench 4.1 is here with updated verifiers for all tasks. In this release, we focused on: - resisting reward hacking - reducing false negatives - improving reproducibility Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping. Leaderboard scores remain within previously reported confidence intervals.

Post attachment
Steven Dillmann avatar
Steven Dillmann@StevenDillmann·Nov 5

Congrats to @bingran_bry & co. on FrontierPhysics 🎉⚛️ Another strong AI-for-science eval, this time for end-to-end physics research. Great to see more benchmarks built on real research workflows.

Post attachment
Yue avatar
⟳ Steven Dillmann retweeted
Yue@Yue1650749·Nov 4

siderius.scitra.org/hackatho… 🚀 The Galileo Hackathon: Track 1 is OPEN! Bring your curiosity and turn your scientific questions into AI for Science challenges. Explore solutions together in Track 2, with opportunities to develop promising ideas into full research projects and papers. 🔬 Expert Scientific Committee across multiple disciplines 
 📖 Students especially welcome—no prior AI or coding experience needed! #AIforScience #hackathon #UCSD

Post attachment
alphaXiv avatar
⟳ Steven Dillmann retweeted
alphaXiv@askalphaxiv·Oct 30

Artificial Analysis Intelligence Index v5.0 came out two days ago. Not only did Terminal-Bench-Science make its debut in the index (15% of the total index), but the entire “Science” category was also up-weighted from 20% to 25%. Scientific research is more important than ever, but researchers aren’t always using the most intelligent models to power their research agents. Some interesting trends: - Anthropic and OpenAI models still dominate the bulk of usage, making up the 6 of the 7 most popular models - Deepseek V4.1 Flash is the most popular open and Chinese model by a wide margin. Its relatively cheap intelligence is quite popular among researchers. - GPT-6 Luna’s usage punches furthest above its intelligence score while Haiku 5.5 is most underused for its intelligence - Despite recent splashes, Google has yet to make meaningful inroads with its latest Gemini 4 models. It would probably help to make the models usable on launch day instead of making researchers wait 2 weeks.

Post attachmentPost attachment
Yue avatar
⟳ Steven Dillmann retweeted
Yue@Yue1650749·Oct 30

What could you discover with AI agents on your team? Join the Galileo Hackathon at UC San Diego HDSI 🧪 Track 1, Nov 4-18: design a scientific ML task ⚡ Track 2, Nov 21: solve fresh tasks with SIDERIUS 🏆 Awards in both tracks, GPU & API credits provided Open to US universities & labs 👇 siderius.scitra.org/hackatho…

2:35