We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%.
1/n 👇
Shout-out to Terminal Bench Science for placing #2! 🎉 It’s an amazing project led by @StevenDillmann that upholds top rigour, the highest standards, and realistic scientific tasks for frontier agents to benchmark on. congrats to the team on the ongoing recognition! 👏
We are currently building the next version of Terminal-Bench.
Creating a great task is very hard but worth the effort as Jason points out: a task sets a hill to climb which large amounts of resources are directed towards.
If you want models to get better at a capability that you care about, create a task.
We work 1-1 with authors from proposal to release. If you are at the top of your field or know someone who is and wants to participate in making Terminal-Bench, DM me!
A room full of AI researchers, developers, and engineers. The launch of Intelligence Index v5. A look at where AI evaluation goes next.
Highlights from our launch event in San Francisco.
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.
Congrats to @bingran_bry & co. on FrontierPhysics 🎉⚛️
Another strong AI-for-science eval, this time for end-to-end physics research. Great to see more benchmarks built on real research workflows.
siderius.scitra.org/hackatho…
🚀 The Galileo Hackathon: Track 1 is OPEN!
Bring your curiosity and turn your scientific questions into AI for Science challenges. Explore solutions together in Track 2, with opportunities to develop promising ideas into full research projects and papers.
🔬 Expert Scientific Committee across multiple disciplines
📖 Students especially welcome—no prior AI or coding experience needed!
#AIforScience#hackathon#UCSD
Artificial Analysis Intelligence Index v5.0 came out two days ago. Not only did Terminal-Bench-Science make its debut in the index (15% of the total index), but the entire “Science” category was also up-weighted from 20% to 25%.
Scientific research is more important than ever, but researchers aren’t always using the most intelligent models to power their research agents. Some interesting trends:
- Anthropic and OpenAI models still dominate the bulk of usage, making up the 6 of the 7 most popular models
- Deepseek V4.1 Flash is the most popular open and Chinese model by a wide margin. Its relatively cheap intelligence is quite popular among researchers.
- GPT-6 Luna’s usage punches furthest above its intelligence score while Haiku 5.5 is most underused for its intelligence
- Despite recent splashes, Google has yet to make meaningful inroads with its latest Gemini 4 models. It would probably help to make the models usable on launch day instead of making researchers wait 2 weeks.
What could you discover with AI agents on your team?
Join the Galileo Hackathon at UC San Diego HDSI
🧪 Track 1, Nov 4-18: design a scientific ML task
⚡ Track 2, Nov 21: solve fresh tasks with SIDERIUS
🏆 Awards in both tracks, GPU & API credits provided
Open to US universities & labs 👇
siderius.scitra.org/hackatho…