A talk on verifiable environments for agents in biology
0:00 Where does experimental data come from?
1:41 Modern bio research organized around measurement
2:44 Data analysis is an executable substrate for science
3:23 Five years of learnings building products for pharma
4:13 Trying to use coding models for biology workflows
5:40 Why frontier models can't be trusted (yet)
6:44 Sequencing based spatial analysis
7:38 SpatialBench design principles
8:26 Anatomy of a single evaluation
9:40 Human verification and early long horizon extensions
13:12 Grading functions and the case against rubrics
14:07 New work in multi-omics, therapeutics, biosecurity
Strongly believe building new wet lab infra in SF is investor catnip. There are many problems with biotech but the market for generating (quite sophisticated) datasets for cash is low on that list.
Cycle through the WPT (“WuXi/Pharmaron Test”):
- Is the assay something WuXi/Pharmaron can run?
- If so, is the team equipped to run it at the quality, price and speed of W/P? This is very difficult.
- If all of the above, can the assay change shape with minimal additional build out/time/expertise? Or are they accidentally building something with a fixed shape and the economics and pace of a platform biotech?
Think these differences in focus and capital allocation will show up clearly in growth over the next 6 months.
Following our release of NeuroscienceBench from @LatchBio last week, we are proud to share a short video walkthrough and a series of example evaluations on benchmarks.bio - we hope that our work is informative to the many neuroscientists gathered at #SfN2026 this week as they discuss the utility of agentic AI in their workflows.
Occasionally I take a look at #general and wonder who all these people are. Latch has grown so much so fast and is only growing faster. Throughout this growth it has stayed a place where a love for science and an awareness of how science can improve lives has stayed constant.
To support these efforts, we're expanding the MTS team across chemistry, assay development, and clinical science, with roles open in:
- MTS, Chemistry
- MTS, Medicinal Chemistry
- MTS, Molecular Design
- MTS, Experimental Biology + Assay Development
- MTS, Translational Science
- MTS, Clinical Development // Clinical Operations // Clinical Data // Trial Design
Some Latch shill on where we've had an impact:
- GDPx biology: gdpx.semianalysis.com/domain…
- Fireworks SII: fireworks.ai/specialized-int…
- Gemini 4 Hydrogen: blog.google/innovation-and-a…
- Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6…
- Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44…
- Grok 4.7 model card: media.x.ai/v1/website/card4p…
- xAI Biosecurity report: x.ai/news/biosafety-at-the-f…
Apply at jobs.ashbyhq.com/latchbio. DMs open
We are growing at a breakneck clip and have ambitious engineering goals in therapeutics. Expanding our MTS team across chemistry, assay dev, translation and the clinic:
- MTS, Chemistry
- MTS, Medicinal Chemistry
- MTS, Molecular Design
- MTS, Experimental Biology + Assay Development
- MTS, Translational Science
- MTS, Clinical Development
- MTS, Clinical Operations
- MTS, Clinical Data + Statistical Programming
- MTS, Biostatistics + Trial Design
High-intensity, founder-type roles. Translate a deep understanding of a biological domain into a roadmap of benchmarks, then build and manage your own scientific team. Our MTS have built companies, run drug programs and scaled complex technical projects.
In person in Mission Bay, San Francisco. We keep meetings to a minimum. Tight deadlines and customer commitments sometimes require late hours.
350K–500K+ OTE + equity. Visa sponsorship and relocation assistance. Our process is typically completed within two weeks.
Frontier AI labs/pharma use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including:
- GDPx biology: gdpx.semianalysis.com/domain…
- Fireworks SII: fireworks.ai/specialized-int…
- Gemini 4 Hydrogen: blog.google/innovation-and-a…
- Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6…
- Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44…
- Grok 4.7 model card: media.x.ai/v1/website/card4p…
- xAI Biosecurity report: x.ai/news/biosafety-at-the-f…
Apply at jobs.ashbyhq.com/latchbio. Or send an email to kenny[at]latch[dot]bio.
Zane is a rare blend of deep scientific thinker and strong engineer - emblematic of the technical leadership we're building at Latch. Beyond stoked to welcome him to the team.
a new way to contribute to scientific progress is emerging
until recently, there were effectively two paths for scientists:
- stay in academia: get to work on more or less whatever you want, but be resource limited and underpaid
- go into industry: be well resourced and well paid, but have to work on narrow, research programs that increase shareholder value
there’s now a third option: contributing to improving the scientific capabilities of AI. this has some of the best aspects of both traditional industry and academia
- you get freedom to pursue the research that interests you
- your work is very high leverage, helping accelerate research of all scientists who use AI
- you are resourced and paid much better than academia
interestingly, what makes it possible for this third path to exist is that the frontier labs actually mean it when they say they want to “cure all disease in 5 years.” whether or not you think this is plausible, there is strong conviction and abundant capital trying to make it happen
i’ve joined @latchbio and i’m very excited to be part of this new wave of scientists striving to ensure AI can contribute to revolutionary scientific discovery
@latchbio we build benchmarks and evaluations that frontier labs and pharma use to measure and improve their models
i am building out a team focused on Aging & Longevity whose goal is to give AI (1) a deep understanding of aging biology, (2) the ability to develop next generation biomarkers of aging, and (3) the capability to discover and develop anti-aging interventions.
if you’re interested in helping accelerate research across the aging & longevity ecosystem my DMs are open
latch.bio/careers/ai-biologi…
a new way to contribute to scientific progress is emerging
until recently, there were effectively two paths for scientists:
- stay in academia: get to work on more or less whatever you want, but be resource limited and underpaid
- go into industry: be well resourced and well paid, but have to work on narrow, research programs that increase shareholder value
there’s now a third option: contributing to improving the scientific capabilities of AI. this has some of the best aspects of both traditional industry and academia
- you get freedom to pursue the research that interests you
- your work is very high leverage, helping accelerate research of all scientists who use AI
- you are resourced and paid much better than academia
interestingly, what makes it possible for this third path to exist is that the frontier labs actually mean it when they say they want to “cure all disease in 5 years.” whether or not you think this is plausible, there is strong conviction and abundant capital trying to make it happen
i’ve joined @latchbio and i’m very excited to be part of this new wave of scientists striving to ensure AI can contribute to revolutionary scientific discovery
@latchbio we build benchmarks and evaluations that frontier labs and pharma use to measure and improve their models
i am building out a team focused on Aging & Longevity whose goal is to give AI (1) a deep understanding of aging biology, (2) the ability to develop next generation biomarkers of aging, and (3) the capability to discover and develop anti-aging interventions.
if you’re interested in helping accelerate research across the aging & longevity ecosystem my DMs are open
latch.bio/careers/ai-biologi…
Think the problem with p-values is simply that almost no one understands them. They require as much coursework as something like Kalman filters, but get cargo culted as early as high school in biology. The formal understanding of eg. power most would dismiss probably directly solves the problem described: if you can whiteboard the calculation, you can see how absurd the gaps between the statistical device and the real world are.
A full generation of young scientists have grown up thinking the p-value is the way to measure scientific rigor. That is a collective failure of our scientific training mission. It's much easier to teach, but it has had a very negative effect on how young scientists think.
New agentic benchmark covering vaccines and other treatments for respiratory viruses.
Prior work in respiratory surveillance focused on detecting and tracking threats. BioSecBench-RespiratoryVirus-Countermeasures (will work on our names) extends task distribution to understanding + treating disease.
64 evaluations cover the following groups:
- viral structure and function
- immunity and antigenic variation
- host response and disease biology
- antiviral activity and resistance
- clinical and population outcomes.
There are three public tasks for inspection spanning antibody occupancy, neutralization of Omicron and vaccine safety reporting.
Gemini 4 Carbon with Pi had the highest overall pass rate at 30.2%, followed by Claude Sonnet 5.5 with Pi at 18.8% and Claude Fable 5.2 with Claude Code at 17.7%. Refusals and errors counted as failures.
Note: Low scores were actually not solely due to refusals. eg. clinical and population outcome tasks had no refusals
Strong work from @harm0n , @evanseeyave, Gage Moreno, Yuyi Liu, and the biosecurity team.
- Leaderboard: benchmarks.bio/respvirus-cou…
- Paper: latch.bio/biosecbench-respvi…
- GitHub: github.com/latchbio/biosecbe…
- Blog: blog.latch.bio/p/can-ai-agen…
Frontier LLMs as:
- simulators of humans interpreting biology
- direct simulators of biology
The first will see faster development + diffusion:
(1) The general intelligence of frontier, arguably even open weight models, has long exceeded the threshold needed to reproduce top percentile human decisions given high quality context/evals. While biology has more complex objects of study than math/software, reproducing human interpretation of these objects does not require knowing all their hidden rules. This work will fall when the conditions are appropriately specified.
(2) Agents deployed this way fit into the existing structure of industry, replacing human interpretation at junctions between experiments that remain bound by physical processes.
The second receives disproportionate funding, attention, energy. Likely technically possible - baking experimental measurements into weights at sufficient scale, especially as assays improve, might accidentally induce models of pathways, cell–cell interactions, etc. needed to generalize. Even vanilla scaled transformers might become something approaching the idealized “virtual cell.” But this requires significantly more time.
There is a lot of technical work left in the assays and data generating platforms themselves. The measurements need to provide sufficient information about underlying dynamics for models to generalize. More measurements are useful to the extent that they actually contain this information. Good take here: x.com/suragnair/status/21072…
The industry structure matters here too. If you can directly model biology well enough to predict experimental outcomes, the incentive is to use this for development yourself. Capture value by owning larger sections of programs before licensing/partnering. Slowly expand program scope, eventually dosing patients yourself.
This means building companies that take on more of the experimental work, capital requirements, and clinical risk. Which takes *time*. While inspiring and likely the more effective technology at the limit, there is no coding agent shaped growth bridging the valley of despair when models are built in the second way.
Think many in our domain don’t yet fully appreciate that frontier models will first emulate how humans reason about biology rather than simulate biological phenomena directly.
Scientists already create enormous value by interpreting imperfect experimental data. They do this with incomplete knowledge of the underlying biology. There are decades of historical experimental work from which to teach and evaluate this judgment.
Scaling that reasoning carries far less technical risk and more immediate economic value than faithfully simulating biological systems. We expect this to restructure drug development over the next few years in much the same way these models are reshaping software engineering and other white collar work.
Excited about direct simulation, and working on it in a calibrated way, but important to be realistic about diffusion and local technical opportunity.
Gemini 4 Hydrogen is a strong scientific model, consistently a top performer across discovery and development benchmarks in a variety of modalities, including antibodies, oligonucleotides and small molecules. Very competitive with the usual suspects.
Pictured is antibody development: benchmarks.bio/txbench-abdev
Proud to see scientists from our biologics development team featured in the Gemini 4 Hydrogen launch.
If you're interested in working on frontier biology evaluation, where engineers and scientists work side by side to improve these systems, shoot me an email at kenny[at]latch[dot]bio.
blog.google/innovation-and-a…
Introducing BioSecBench-Structure, a structural reasoning benchmark for biosecurity tasks.
We measure agentic interpretation of molecular structures to assess pathogen targets and support the development of vaccines, antibodies, and drugs.
Ex:
- Coronavirus spike conformations. Nine cryo-EM structures capture receptor-binding domains in open and closed states, which affect receptor engagement and antibody access. Agents determine how many domains are open in each structure
- Inhibitor screening against a bacterial pathogen. Two structures of a protein involved in pilus assembly are being considered as templates for screening inhibitors. Agents compare the protein’s subunits to determine whether screening can use a single structural template or needs an ensemble.
- Candidate drugs targeting a Pseudomonas aeruginosa enzyme. Two compounds differ by a small chemical modification. Agents examine crystal structures and electron-density maps to determine whether the modification increases the number of occupied binding sites before it is carried into the next design round.
The benchmark comprises 58 evaluations built from crystal and cryo-EM structures, predicted models, and experimental measurements. Thirty-three tasks concern threat characterization, 22 concern countermeasure design, and three concern threat detection.
We tested 20 models in 35 model–harness configurations, with three offline runs per evaluation. With refusals enabled, Claude Fable 5.2 with Claude Code had the highest overall pass rate at 43.7%, followed by Fable 5.2 with Pi at 41.4% and Claude Opus 5.5 with Claude Code at 31.0%. Refusals and errors counted as failures.
The leaderboard also includes Anthropic’s safety-filter-exempt evaluation variants (LSVP). Fable 5.2 LSVP with Pi reached 52.3%, and Opus 5.5 LSVP with Pi reached 39.7%.
Refusals concentrated on defensive countermeasure work. In reruns with refusals disabled, countermeasure pass rates increased by 14 percentage points for Fable 5.2 and 15 for Opus 5.5, with harnesses pooled. Their threat-characterization scores barely changed.
We think structural reasoning will be an important capability for agents supporting biosecurity. Important work from John Wang, Nick Keul, Jonathan Felix, Qian Xu, and team.
Benchmarks[dot]bio: benchmarks.bio/biosec-struct…
Paper: latch.bio/biosecbench-struct…
GitHub: github.com/latchbio/biosecbe…
Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.
Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.
The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.
Some of the work:
- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.
Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.
Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.
We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.
We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.
We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.
- Blog: blog.latch.bio/p/tissue-micr…
- GitHub: github.com/latchbio/tissuemi…
- Leaderboard: benchmarks.bio/tissuemicrosc…