At GitHub Universe in October, I talked about multi-model solutions like Copilot Fusion will start dominating benchmarks going forward. But the pace of progress has been faster - ~10% better quality at half cost just few weeks later.
The old way: users are human, you run user interviews, get inspired, mock up user journeys.
The new way: users are agents, you run evals, evals inform the software engineering loop.
Good read on how we do evals at Vercel.
Your next users are agents. Evals are how you build for them.
▪︎ See how agents use your product
▪︎ Test changes before you ship
▪︎ Compare models as they launch
vercel.com/blog/agent-evals
Given that there are only 66 problems in TerminalBench v4.1 and literally every frontier lab tracks TerminalBench performance, at some point it will be more prestigious and impactful to have authored a TerminalBench problem than to get a paper accepted in Nature or Science
Introducing FrontierBioML: 25 long-horizon machine learning tasks in computational biology with 11 models, up to 6 hours per run. Agents submit a prediction script; we run it on a held-out test set.
Claude Mythos 5.2 leads (40.3), then Gemini 4 Carbon (39.0) and Gemini 4 Argon (36.8)
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎!
Rollout servers are APIs / MCPs that you (or your agent) to
Launch jobs
Monitor rollouts
Query results
Pause, cancel, retry, or resume jobs and rollouts
They're the missing control plane to Harbor workloads.
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎 which turns harbor into a server lets you submit jobs, query/pause/resume/cancel jobs and rollouts, and retry individual rollouts.
It's a huge unlock for RL and one of the most elegant and powerful features Alex and I built in harbor.
four ways to connect:
1. python sdk: Harbor() / AsyncHarbor()
2. cli: harbor run --server http://localhost:8390
3. mcp: /mcp
4. http: POST /jobs/submit
so much more control and observability unlocked:
docs.harborframework.com/har…docs.harborframework.com/api…
Introducing 𝚑𝚊𝚛𝚋𝚘𝚛 𝚜𝚎𝚛𝚟𝚎!
Rollout servers are APIs / MCPs that you (or your agent) to
Launch jobs
Monitor rollouts
Query results
Pause, cancel, retry, or resume jobs and rollouts
They're the missing control plane to Harbor workloads.
Agents can write code. Can they finish real engineering work?
Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.
Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.
OWE-bench top 3 leaderboard:
1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.
Explore the benchmark website: openworldengineer.mercor.com…
Biopharma Bench V0.2 is a huge upgrade from the previous version, mitigating eval awareness with the @harborframework team and working with experts at Lonza, Catalent, Takeda and Sanofi to audit the rubrics and expert judgment. Check it out!
🚀 SWE-rebench V2.1 RL is out: 13,337 free RL environments for coding agents (20 languages, Harbor format)!
🔧 Fixed tasks, envs, and graders → fast, reliable rollouts.
Doing RL post-training or research? DM or reply. We can share data and RL envs early for free, so we can make them better for open source!
🙏 Please RT or share with anyone who might need free RL envs!