←
harmon500 posts
harmon avatar

harmon

@harm0n

biosecurity lead @latchbio

◷ Joined May 2021▧ 12 Photos and videos
harmon avatar
harmon@harm0n·3 Nov 2025

Ever wonder how you could use your computer science skills to stop the next pandemic, but don't know where to start. In a new post, I provide an introduction into viral biosurveillance and the computational problems that come with it.

harmon avatar
harmon@harm0n·21h

listening to pokémon music while working is tech

Christopher Zou avatar
⟳ harmon retweeted
Christopher Zou@chriswzou·Nov 17

Occasionally I take a look at #general and wonder who all these people are. Latch has grown so much so fast and is only growing faster. Throughout this growth it has stayed a place where a love for science and an awareness of how science can improve lives has stayed constant. To support these efforts, we're expanding the MTS team across chemistry, assay development, and clinical science, with roles open in: - MTS, Chemistry - MTS, Medicinal Chemistry - MTS, Molecular Design - MTS, Experimental Biology + Assay Development - MTS, Translational Science - MTS, Clinical Development // Clinical Operations // Clinical Data // Trial Design Some Latch shill on where we've had an impact: - GDPx biology: gdpx.semianalysis.com/domain… - Fireworks SII: fireworks.ai/specialized-int… - Gemini 4 Hydrogen: blog.google/innovation-and-a… - Claude Fable 5.1/Mythos 5.1 system card: www-cdn.anthropic.com/0339e6… - Claude Opus 5.5 system card: www-cdn.anthropic.com/fc1b44… - Grok 4.7 model card: media.x.ai/v1/website/card4p… - xAI Biosecurity report: x.ai/news/biosafety-at-the-f… Apply at jobs.ashbyhq.com/latchbio. DMs open

Jack Ziyang Cai avatar
⟳ harmon retweeted
Jack Ziyang Cai@jackcai1206·Nov 16

New paper w/ Yihe Dong, @DimitrisPapail & @ShivamGarg91462: Perturbing Inputs Improves Pretraining Distillation. In pretraining distillation, a small student LM learns to match a larger teacher’s next-token predictions on the training text. We change one thing: for the first part of training, both models read a perturbed version of the text (for example, with noise added to the token inputs), and the student matches the teacher’s predictions on that. Then we switch back to clean text. With the same steps and token budget, this beats standard distillation, and it also works well in data-constrained, multi-epoch settings. 🧵

Post attachment
Zane Koch avatar
⟳ harmon retweeted
Zane Koch@zanehkoch·Nov 15

a new way to contribute to scientific progress is emerging until recently, there were effectively two paths for scientists: - stay in academia: get to work on more or less whatever you want, but be resource limited and underpaid - go into industry: be well resourced and well paid, but have to work on narrow, research programs that increase shareholder value there’s now a third option: contributing to improving the scientific capabilities of AI. this has some of the best aspects of both traditional industry and academia - you get freedom to pursue the research that interests you - your work is very high leverage, helping accelerate research of all scientists who use AI - you are resourced and paid much better than academia interestingly, what makes it possible for this third path to exist is that the frontier labs actually mean it when they say they want to “cure all disease in 5 years.” whether or not you think this is plausible, there is strong conviction and abundant capital trying to make it happen i’ve joined @latchbio and i’m very excited to be part of this new wave of scientists striving to ensure AI can contribute to revolutionary scientific discovery @latchbio we build benchmarks and evaluations that frontier labs and pharma use to measure and improve their models i am building out a team focused on Aging & Longevity whose goal is to give AI (1) a deep understanding of aging biology, (2) the ability to develop next generation biomarkers of aging, and (3) the capability to discover and develop anti-aging interventions. if you’re interested in helping accelerate research across the aging & longevity ecosystem my DMs are open latch.bio/careers/ai-biologi…

harmon avatar
harmon@harm0n·Nov 13

Another benchmark from our group! We're working on our benchmark names 😅, if you have any recommendations, drop them below.

New agentic benchmark covering vaccines and other treatments for respiratory viruses. Prior work in respiratory surveillance focused on detecting and tracking threats. BioSecBench-RespiratoryVirus-Countermeasures (will work on our names) extends task distribution to understanding + treating disease. 64 evaluations cover the following groups: - viral structure and function - immunity and antigenic variation - host response and disease biology - antiviral activity and resistance - clinical and population outcomes. There are three public tasks for inspection spanning antibody occupancy, neutralization of Omicron and vaccine safety reporting. Gemini 4 Carbon with Pi had the highest overall pass rate at 30.2%, followed by Claude Sonnet 5.5 with Pi at 18.8% and Claude Fable 5.2 with Claude Code at 17.7%. Refusals and errors counted as failures. Note: Low scores were actually not solely due to refusals. eg. clinical and population outcome tasks had no refusals Strong work from @harm0n , @evanseeyave, Gage Moreno, Yuyi Liu, and the biosecurity team. - Leaderboard: benchmarks.bio/respvirus-cou… - Paper: latch.bio/biosecbench-respvi… - GitHub: github.com/latchbio/biosecbe… - Blog: blog.latch.bio/p/can-ai-agen…

Post attachment
harmon avatar
harmon@harm0n·Nov 12

Ever wonder how agents do at structural reasoning in biosecurity, well now you don't have to!!

Introducing BioSecBench-Structure, a structural reasoning benchmark for biosecurity tasks. We measure agentic interpretation of molecular structures to assess pathogen targets and support the development of vaccines, antibodies, and drugs. Ex: - Coronavirus spike conformations. Nine cryo-EM structures capture receptor-binding domains in open and closed states, which affect receptor engagement and antibody access. Agents determine how many domains are open in each structure - Inhibitor screening against a bacterial pathogen. Two structures of a protein involved in pilus assembly are being considered as templates for screening inhibitors. Agents compare the protein’s subunits to determine whether screening can use a single structural template or needs an ensemble. - Candidate drugs targeting a Pseudomonas aeruginosa enzyme. Two compounds differ by a small chemical modification. Agents examine crystal structures and electron-density maps to determine whether the modification increases the number of occupied binding sites before it is carried into the next design round. The benchmark comprises 58 evaluations built from crystal and cryo-EM structures, predicted models, and experimental measurements. Thirty-three tasks concern threat characterization, 22 concern countermeasure design, and three concern threat detection. We tested 20 models in 35 model–harness configurations, with three offline runs per evaluation. With refusals enabled, Claude Fable 5.2 with Claude Code had the highest overall pass rate at 43.7%, followed by Fable 5.2 with Pi at 41.4% and Claude Opus 5.5 with Claude Code at 31.0%. Refusals and errors counted as failures. The leaderboard also includes Anthropic’s safety-filter-exempt evaluation variants (LSVP). Fable 5.2 LSVP with Pi reached 52.3%, and Opus 5.5 LSVP with Pi reached 39.7%. Refusals concentrated on defensive countermeasure work. In reruns with refusals disabled, countermeasure pass rates increased by 14 percentage points for Fable 5.2 and 15 for Opus 5.5, with harnesses pooled. Their threat-characterization scores barely changed. We think structural reasoning will be an important capability for agents supporting biosecurity. Important work from John Wang, Nick Keul, Jonathan Felix, Qian Xu, and team. Benchmarks[dot]bio: benchmarks.bio/biosec-struct… Paper: latch.bio/biosecbench-struct… GitHub: github.com/latchbio/biosecbe…

Post attachment
Kenny Workman avatar
⟳ harmon retweeted
Kenny Workman@kenbwork·Nov 11

Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science. Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems. The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time. Some of the work: - Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion. - Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images. - Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries. - A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration. Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning. Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality. We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration. We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult. We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here. - Blog: blog.latch.bio/p/tissue-micr… - GitHub: github.com/latchbio/tissuemi… - Leaderboard: benchmarks.bio/tissuemicrosc…

0:17
Kenny Workman avatar
⟳ harmon retweeted
Kenny Workman@kenbwork·Nov 9

We’re expanding our benchmark efforts into basic research. Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works. We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment. NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically. Three public examples: - Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff. - Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation. - A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters. We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained. Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%. Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with. Paper: latch.bio/neurobench Leaderboard: benchmarks.bio/neuroscience Blog: blog.latch.bio/p/neuroscienc… GitHub: github.com/latchbio/neuroben… Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.

Post attachment
Christopher Zou avatar
⟳ harmon retweeted
Christopher Zou@chriswzou·Nov 6

(3/4 releases) Today we share RareDiseaseBench, a series of 35 diagnostic workups reconstructed from published rare disease cases. In each task, an agent starts with a patient’s presentation, decides which tests to request and why, analyzes the data it receives, and uses the results to propose a diagnosis and management plan over one or more visits. Gemini 4 Argon does the best on this benchmark (43.8% attempt pass rate), with Opus 5.5 and Astra following. Paper: latch.bio/raredisease Results: benchmarks.bio/raredisease More details in thread:

Post attachment
Gage Moreno avatar
⟳ harmon retweeted
Gage Moreno@GageKMoreno·Nov 5

You know I love a respiratory virus 👀

AI will become core infrastructure for detecting and tracking respiratory outbreaks. Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance. Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures. Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal. This benchmark was an especially large effort from many talented scientists. Leaderboard: benchmarks.bio/respvirus-sur… Blog: blog.latch.bio/p/biosecbench… Paper: latch.bio/respvirus-surveill… GitHub: github.com/latchbio/biosecbe…

Post attachment
harmon avatar
harmon@harm0n·Nov 5

If you've worked on medical countermeasures, come work with us we're hiring! (link below)

AI will become core infrastructure for detecting and tracking respiratory outbreaks. Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance. Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures. Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal. This benchmark was an especially large effort from many talented scientists. Leaderboard: benchmarks.bio/respvirus-sur… Blog: blog.latch.bio/p/biosecbench… Paper: latch.bio/respvirus-surveill… GitHub: github.com/latchbio/biosecbe…

Post attachment
harmon avatar
harmon@harm0n·Nov 5

One of several big releases coming from the biosecurity group in the coming weeks.

AI will become core infrastructure for detecting and tracking respiratory outbreaks. Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance. Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures. Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal. This benchmark was an especially large effort from many talented scientists. Leaderboard: benchmarks.bio/respvirus-sur… Blog: blog.latch.bio/p/biosecbench… Paper: latch.bio/respvirus-surveill… GitHub: github.com/latchbio/biosecbe…

Post attachment
harmon avatar
harmon@harm0n·Nov 5

Biosecurity at Latch is cracked!

AI will become core infrastructure for detecting and tracking respiratory outbreaks. Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance. Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures. Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal. This benchmark was an especially large effort from many talented scientists. Leaderboard: benchmarks.bio/respvirus-sur… Blog: blog.latch.bio/p/biosecbench… Paper: latch.bio/respvirus-surveill… GitHub: github.com/latchbio/biosecbe…

Post attachment