←
Generality Labs18 posts
Generality Labs avatar

Generality Labs

@GeneralityLabs
◷ Joined September 2026▧ 12 Photos and videos
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 30

We see clear diminishing returns as teams get bigger. Adding more agents can slow the team down and lower its score. This could be because of failures in the ability to effectively coordinate in larger teams of agents.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 30

We gave the agents a message board, as shown in the video below. They could post to a global channel or send direct messages to each other. How they collaborated and delegated tasks was entirely emergent. We gave them no rules on how to use the message board.

1:33
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 30

Collaborating agents can complete work faster and sometimes do it better. We ran experiments on MirrorCode with teams of 1, 2, 4, 8, 16, 32 and 64 agents. On average, teams completed tasks much faster and passed more unit tests than single agents.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

Recommendations: • Report LABBench2 per task set, not as one aggregate • Drop the saturated sets (TrialQA, PatentQA, SourceQuality, TableQA2, maybe LitQA3) unless checking for regression • Fix harness for model comps, especially for DbQA2 and SuppQA2

Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

We generally like CloningQA, which requires compliance with a custom workflow (relevent!) The main issue is it only has 14 tasks, so it’s tough to get much insight. We’d like to see more task variations.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

<Aside> Most of these tasks also implicitly require web search but never state this. As a result, well behaved models like Claude look much worse because they don’t want to “cheat” by looking the papers up online.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

Most of LB2 is built around literature tasks, and most of those are saturated. The only literature task with major headroom is FigQA2, but this is largely image processing. So, most of the “literature” improvement will actually just track reading figures.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

DbQA2 is also much easier than it looks because the default harness is weak. Using an agentic harness almost doubles performance. Since most comp bio is done in something like Claude code rather than a chat box, DbQA2 is probably close to saturation.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

Almost every task has some issues with underspecification. DbQA2 is likely the worst, with ~40% of failures coming from unclear prompts w/ multiple interpretations. Example: Asked to find a transcription factor, but there are 7 valid variants.

Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

The choice of judge model also has a big impact on ProtocolQA2 scores (probably because the solution is underdetermined). Swapping GPT-5.6 Terra to Gemini 3.7 Flash changes the scores on the same answer by ~12%.

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

ProtocolQA2 has poor construct validity. Agents are asked to solve a problem with a failing protocol, but there are often multiple valid solutions and only one is accepted. This works as a multiple choice, but not as a free response.

Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

SeqQA2 has enough issues that we wrote a whole separate post about it. If you correct for those it’s saturated. Also, be careful using it as 100 tasks nuke your agent with a 500k+ token input. This can get expensive (or fail if your budget is small).

Post attachment
Generality Labs avatar
Generality Labs@GeneralityLabs·Oct 9

LABBench2 has been used by several labs and other orgs to measure the bio-capabilities of frontier models. We audited it and found some issues: many task sets are saturated, two are flawed, and the rest are at least moderately concerning. More below🧵

Post attachment