




We see clear diminishing returns as teams get bigger. Adding more agents can slow the team down and lower its score. This could be because of failures in the ability to effectively coordinate in larger teams of agents.

Every team had the same 250M token budget, split equally between its agents.

We gave the agents a message board, as shown in the video below. They could post to a global channel or send direct messages to each other. How they collaborated and delegated tasks was entirely emergent. We gave them no rules on how to use the message board.

Collaborating agents can complete work faster and sometimes do it better. We ran experiments on MirrorCode with teams of 1, 2, 4, 8, 16, 32 and 64 agents. On average, teams completed tasks much faster and passed more unit tests than single agents.

More in the full post, including gated transcript examples: generality.org/blog/posts/la… And the SeqQA2 post: generality.org/blog/posts/la…

Recommendations: • Report LABBench2 per task set, not as one aggregate • Drop the saturated sets (TrialQA, PatentQA, SourceQuality, TableQA2, maybe LitQA3) unless checking for regression • Fix harness for model comps, especially for DbQA2 and SuppQA2

We generally like CloningQA, which requires compliance with a custom workflow (relevent!) The main issue is it only has 14 tasks, so it’s tough to get much insight. We’d like to see more task variations.

<Aside> Most of these tasks also implicitly require web search but never state this. As a result, well behaved models like Claude look much worse because they don’t want to “cheat” by looking the papers up online.

Most of LB2 is built around literature tasks, and most of those are saturated. The only literature task with major headroom is FigQA2, but this is largely image processing. So, most of the “literature” improvement will actually just track reading figures.

DbQA2 is also much easier than it looks because the default harness is weak. Using an agentic harness almost doubles performance. Since most comp bio is done in something like Claude code rather than a chat box, DbQA2 is probably close to saturation.

Almost every task has some issues with underspecification. DbQA2 is likely the worst, with ~40% of failures coming from unclear prompts w/ multiple interpretations. Example: Asked to find a transcription factor, but there are 7 valid variants.

The choice of judge model also has a big impact on ProtocolQA2 scores (probably because the solution is underdetermined). Swapping GPT-5.6 Terra to Gemini 3.7 Flash changes the scores on the same answer by ~12%.

ProtocolQA2 has poor construct validity. Agents are asked to solve a problem with a failing protocol, but there are often multiple valid solutions and only one is accepted. This works as a multiple choice, but not as a free response.

SeqQA2 has enough issues that we wrote a whole separate post about it. If you correct for those it’s saturated. Also, be careful using it as 100 tasks nuke your agent with a 500k+ token input. This can get expensive (or fail if your budget is small).

LABBench2 has been used by several labs and other orgs to measure the bio-capabilities of frontier models. We audited it and found some issues: many task sets are saturated, two are flawed, and the rest are at least moderately concerning. More below🧵