Discipline gaps: no model wins every field.
Average scores: medicine 0.394, materials 0.378, chemistry 0.294, biology 0.290.
Best per field: Opus 5.5 in chemistry, Fable 5.1 in materials, GPT-6 Astra Medium in biology, Opus 5 in medicine.
Chemistry has the widest spread, leaders about twice the laggards; biology the narrowest.
Shuffle the discipline labels and differences of the same size appear (P = 0.22).
Behavior: how each model spends its eight hours.
About 5% of the run goes to reading the input, search concentrates in the first quarter, computation peaks mid-run, the last tenth writes the submission.
Runtime spent searching: about 30% for GPT-6 Astra Medium and GPT-6.1 Sol, 6% for Claude Opus 5.5. Median searches: about 20 for GPT-6.1 Sol and Kimi-K3.1-Pro, 3 for Claude Sonnet 5.5 and Qwen3.8.
Python failures: about 4% for Opus 5.5, about 17% for Doubao and HY4.
Across 1,155 graded runs and 227,103 tool calls, failures come in four kinds: wrong quantity, wrong target, an unreliable calculation overruling a reasonable expectation, and the shared intuition leading everyone into the same error.
Model capability gaps: six models lead, seven lag.
Same tasks, 0.15 apart. Leaders score below random in 9% of runs; the rest, 22%.
Against a shared intuition built from other models' orderings: where that intuition is right, leaders keep 86 of 100 pairs, laggards 78. Where it is wrong, both fix about 30.
Breaking with consensus: leaders right about half the time, Claude Opus 5.5 at 63%, laggards at 38%.
Three things separate them: when to trust your own calculation, whether the parameter you computed is the quantity being measured, and finding the mechanism when common sense fails. The third varies most.
Models are still far from reliable prediction.
Each task comes from one real study: candidates, conditions, and how the result was measured. The agent orders the comparison groups by the measured quantity.
13 configurations of 12 frontier models. Best score on public tasks: 0.486.
It clears the random threshold on only about half of its graded tasks.
Introducing Science Prediction Universe (SPU).
A benchmark that hides every measurement and asks AI to predict what a real wet-lab experiment will show, before it is run.
100 public tasks built by domain experts from 85 papers in 11 Nature Portfolio journals. 16 private tasks from unpublished wet-lab experiments in partner labs.
Led by UniPat AI, with researchers from Peking University, Stanford, Cornell, Harvard and University of Illinois Chicago, together with Future House, ABclonal, Yurogen, Deep Principle and Bota Bio.
Thanks to every collaborator who took part in this work (a non-exhaustive list, in no particular order): @Dominicliu12@jlwu55@liangchen5518@StevenDillmann@YuanqiD@SoojungYang2@shizhediao@chenru_duan
Blog:spu.unipat.ai
GitHub:github.com/UniPat-AI/SPU
arXiv:arxiv.org/abs/2611.15049
Excited to have worked with @ElorianAI on ScienceVision. Huge thanks to @AndrewDai and everyone who contributed to making this work happen!
It all starts with a question: Can AI observe and think like a scientist?
With ScienceVision, we take this question into the world of scientific 3D objects, exploring whether AI can gather evidence, make sense of what it sees, and reason about it.
More to explore and discover in 3D!
Three evidence-access settings, the same 250 questions:
– Programmatic analysis of 3D source data
– Source access plus Vision MCP inspection tools
– Fixed rendered views, with six additional CT slices for medical imaging and no direct access to 3D source data
Depending on the viewer, tools support changing views, isolating structures, browsing slices and measuring geometry.
Each task specifies an answer format and a reference answer for scoring.
Nearly 500 hours of recorded expert work supported question writing, revision, reference-answer annotation and review.
UniPat AI built and operated the expert platform, coordinating this work across all five disciplines.
Reference inspections show how experts locate targets and gather the evidence needed to answer a question.
Each discipline contributes 50 questions: biology, chemistry, medical imaging, mechanical CAD and architecture.
Tasks include classifying atomic hybridization, locating a lesion in a CT volume and assessing a part’s symmetry.
Models must use domain knowledge to interpret geometric or topological evidence.
ScienceVision combines scientific 3D questions, inspection tools and expert reference answers across disciplines.
All 250 questions have reference inspection records. Harbor-format oracle solutions support task validation and reproducibility.
Watch how experts inspect 3D objects and gather evidence in the reference inspections below.
12 models. 250 scientific 3D questions.
GPT-6 Astra (Codex) leads both 3D access settings, reaching a mean answer score of approximately 0.53 out of 1.
Harness · 3D / Harness · 3D + MCP:
– GPT-6 Astra: 0.53 / 0.53
– GPT-6.1 Sol: 0.49 / 0.51
– Claude Fable 5.1: 0.46 / 0.46
MCP-assisted scores are higher for some models and lower for others. More access does not consistently correspond to higher answer scores.
Scores include partial credit and are rounded to two decimals.
GPT models use Codex; the other models use Claude Code. Results reflect these model–harness configurations.
What is one problem in your field that AI still can’t solve—but should?
We’re launching UniPat Fellows at the UniPat AI Exploration Lab to bring scientific research and model development into the same feedback loop.
Fellows bring hard questions and deep domain expertise. Together with our researchers and engineers, they turn those questions into tasks for evaluating and training models, grounded in specialized tools, simulations, and real-world feedback. As models improve, they can help Fellows advance the research that inspired those tasks.
Our scope spans physics, chemistry, biology, medicine, materials, chip design, and beyond. What matters is whether the problem is important—and whether progress can be tested against observable results.
UniPat AI will invest RMB 100 million over the next year in this effort. Fellows receive:
– Close collaboration with our research and engineering teams
– Frontier-model API credits valued at RMB 500,000
– Compute, specialized data, access to domain experts, and research tools
– Competitive compensation
Two ways to join:
– Full-time Fellow: Join UniPat AI to pursue long-term research and real-world applications.
– Project-based Fellow: Work with us on a project for three months or longer while continuing your academic or professional commitments.
We encourage Fellows to work with us in person in Zhongguancun, Beijing; hybrid arrangements can be discussed.
To apply, email fellows@unipat.ai with your CV, 1–2 selected projects or publications, and the one problem you’d pursue over the next three months—and why.