←
SecureBio243 posts
SecureBio avatar

SecureBio

@SecureBio

SecureBio is a research non-profit that works to advance biotechnology safely and prevent catastrophic pandemics.

◷ Joined January 2025▧ 90 Photos and videos
SecureBio avatar
SecureBio@SecureBio·Nov 16

We are thrilled to announce that Dr. Marisa Donnelly has joined SecureBio Detection as Public Health Response Manager. Dr. Donnelly is an infectious disease epidemiologist who was previously Director of Epidemiology at Biobot Analytics and is a frequent contributor to Your Local Epidemiologist. Before that, she worked on the wastewater team at the California Department of Public Health and served as a CDC Epidemic Intelligence Service Officer and a Lieutenant in the U.S. Public Health Service. Dr. Donnelly will focus on building SecureBio Detection’s relationships with our partners in state, local, territorial, and tribal public health departments. In particular, she will translate the data that our metagenomic sequencing generates into action, to ensure that our public health partners get a clear understanding of what the data mean and to inform what officials might want to do next. “Building an effective response function requires someone who understands not only the science but also the people who are responsible for acting on early warning information. Marisa fits this role perfectly, bringing years of experience from the federal, state, and local level and national-level public health communications. I couldn’t be more excited to welcome her as our first Public Health Response Manager," said Kelly Chafin, who leads the Detection team's response and federal strategy work. "Preventing the next pandemic depends as much on strong relationships built across communities as it does on great technology. And SecureBio is investing in both, which is a big reason I’m so excited to join the team. I can’t wait to jump in and start turning data into action!" Dr. Donnelly said. Welcome to the team, Marisa!

SecureBio avatar
SecureBio@SecureBio·Nov 12

Model cards show how a model’s safeguards performed at launch, but they don’t show what happens afterwards. We’ve begun tracking BioTIER scores over time to see how post-launch adjustments affect model refusals: Claude Opus 5.5’s BioTIER-permit score rose from 37.3% to 56.7% in a recent week (high-risk refusals held steady), and Grok 4.7’s BioTIER-refuse score fell in October. Read our full announcement: securebio.substack.com/p/lon… See all the data on the SecureBio dashboard: securebio.org/benchmarks/det…

Post attachment
SecureBio avatar
SecureBio@SecureBio·Nov 10

We are delighted to announce that Dr. Mike Tisza has joined SecureBio Detection as Principal Scientist. Dr. Tisza was previously an Assistant Professor at Baylor College of Medicine, where he led a microbial genomics research lab as a long-time member of the CASPER consortium (tiszalab.github.io). Dr. Tisza will continue to focus on developing and refining tools for viral and bacterial sequence analysis and pathogen detection, working closely with both the computational and laboratory teams. "Mike is a world expert in pathogen metagenomics and has built many tools that are widely used and respected by the field. I'm thrilled to have him join the team and bring that deep expertise and experience to our detection mission," said Dr. Will Bradshaw, who leads the team's computational work. “I’m extremely excited to join SecureBio because it has a rare combination of amazing colleagues, fascinating science problems, and an important mission. In my opinion, no other organization is working as hard to prevent pandemics, and I’m inspired by the new ideas that the team is coming up with. Coming up with and applying solutions to detect natural and engineered pathogens is something I’m really looking forward to,” Dr. Tisza said. Welcome to the team, Mike!

SecureBio avatar
SecureBio@SecureBio·Nov 6

The 2025 Active Site wet-lab uplift trial found that LLM access didn't help novices with a viral rescue workflow, implying that a lack of tacit knowledge is a key barrier. We and RAND reanalyzed the transcripts and found another explanation—2025 models gave bad advice. Read our full report here: securebio.substack.com/p/202… In 2025, Active Site ran an 8-week randomized controlled wet-lab uplift trial with 153 participants and reported that biology novices with access to LLMs were not significantly better able to complete a viral rescue workflow than novices with internet access only. It’s been argued that these results mean that a lack of tacit knowledge (i.e., hands-on lab expertise) is a key barrier to success in the lab that cannot be overcome by LLM assistance. In collaboration with RAND, we (SecureBio) analyzed the LLM transcripts and lab notebooks from this study and found additional factors that likely limited the uplift observed. In the molecular cloning task, the mid-2025 LLMs used in the study often provided incorrect advice on a purely computational fragment selection step (figuring out which DNA fragments to combine, and which method to use). Of the 55 participants whose transcripts we could score, only 18 (~33%) received correct advice in their first relevant conversation with LLMs. The original analysis reported a high level of success at the fragment selection step, with 56 of the 77 participants in the LLM group ordering the correct DNA fragments. However, we found that many of these participants had actually ordered several of the fragments in the bank, rather than specifically selecting the correct ones. When we looked at the experiments that the participants actually ran, we found that only 29 ever tried the right combination with the right method. There were still failures at the wet-lab steps—only 12 participants ultimately succeeded—but the implied hands-on success rate of 40% (12/29) is much higher than the 21% (12/56) suggested in the original paper. Notably, when we replayed participants’ prompts through newer models, we found that GPT-5.6 Sol gave correct fragment selection advice on 92% of runs. While this doesn't show unambiguously that 2026 models would uplift novices, it does suggest that the models of mid-2025 don’t represent the ceiling of support that LLMs could provide. Future studies should continue to assess the uplift that AI models could provide to actors of different profiles.

Post attachment
SecureBio avatar
SecureBio@SecureBio·Nov 3

In a brief independent assessment, we find that Kimi K3.1 Pro, the latest model from Moonshot AI, is second among open-weight models on our Bio Capabilities Index (BCI) and 19th overall. Its score of 149 places it just behind Kimi K3 (150) and well behind the highest-scoring model, Claude Opus 5 (177). With respect to safeguards, K3.1 Pro refuses only 33.6% of hazardous prompts in our BioTIER benchmark. This is a modest improvement from 26.9% for Kimi K3 but still far behind the classifiers and refusals of closed-weight providers. On our knowledge benchmarks, Kimi K3.1 Pro achieves the top open-weights score on two and improves on Kimi K3 on three of the four benchmarks that we highlight: VCT-v2: 46.2 ± 1.2%, 8th overall MBCT: 60.7 ± 0.9%, 5th overall WCB: 51.5 ± 1.6% (0.9% refused), 27th overall HPCT: 63.5 ± 1.2%, 8th overall We will continue to evaluate new models as they are released and expect to have results for K3.1 Pro on our agentic benchmarks in the coming weeks. Read the full model note: securebio.substack.com/p/kim… Interested in contributing to this kind of work? We’re hiring! See our Careers page for open roles: securebio.org/careers/

Post attachment
SecureBio avatar
SecureBio@SecureBio·Oct 28

A lack of “tacit knowledge” is often treated as the barrier that stops biological misuse. But we think that assumption is doing more work than the evidence supports. Drawing on ~60 years of collective lab experience, we unpack what “tacit knowledge” means and how much of a barrier to harm it actually poses: securebio.substack.com/p/myt… The problem is that one term—“tacit knowledge”—covers a broad spectrum of skills that differ enormously in how hard they are to acquire and whether they’re needed for a dangerous outcome. Pipetting and plating sit in the same bucket as adapting a protocol to a new organism or virus. Lumping basic lab skills and expert judgement together makes the barrier look stronger than it is. Basic lab skills may be all that the simplest paths to harm require, and they are the easy part: training resources are widely available, competence can be achieved in just weeks (in one wet-lab uplift study, ~80% of novices nailed a serial dilution in 4–8 tries), and many protocols tolerate substantial imperfection. It’s still true that novel failures and optimization problems are difficult to overcome without expert judgment. But in assessing biological risk, tacit knowledge may be less of a fortress wall than a collection of low fences that can be hopped with public resources, automation, and, increasingly, AI.

Post attachment
Coleman Breen avatar
⟳ SecureBio retweeted
Coleman Breen@BreenColeman·Oct 26

Very excited to have worked with AISI on our newest agentic benchmark: ReproBAIT. It assesses how well an AI coding agent can reproduce a biological AI tool, and will help generate valuable information on evolving capabilities relevant to biosecurity.

Study 1: Can AI coding agents rebuild a biological AI tool from just its paper? With @SecureBio, we built ReproBAIT to test leading coding agents. For two of three tools, agents reproduced a key function on a single GPU - often for $4–$20.

Post attachment
SecureBio avatar
SecureBio@SecureBio·Oct 26

Thread & blog posts from UK AISI on our recent collaborative efforts: 1) ReproBAIT, a new benchmark to assess general purpose AI model capabilities at reproducing published biological AI tools, and 2) research on the strengths and limitations of pre-training data filtering on the dual-use capabilities of open-weight models.

Study 1: Can AI coding agents rebuild a biological AI tool from just its paper? With @SecureBio, we built ReproBAIT to test leading coding agents. For two of three tools, agents reproduced a key function on a single GPU - often for $4–$20.

Post attachment
SecureBio avatar
SecureBio@SecureBio·Oct 26

In our brief independent assessment, Gemini 4 Argon is the second-highest scoring model on SecureBio's Bio Capabilities Index: 169, behind only Claude Opus 5 (176). On BioTIER, Argon refuses 87.8% of hazardous prompts, up from 62.8% for Gemini 3.1 Pro. Read the full model note here: securebio.substack.com/p/gem… On our knowledge benchmarks, Argon is roughly tied for the top score on VCT-v2 (57.3% vs. 58.0% for GPT-6 Astra), second on MBCT (63.2%), and highest on HPCT (74.2%, though 29.1% of questions were refused). It ranks sixth on WCB. We welcome GDM's substantial safety report. We'd like more detail on how third-party evaluations were run, and more testing of long-horizon agentic capabilities and of uplift for well-resourced groups. Agentic benchmark results for Argon will follow on our dashboard.

Post attachment
SecureBio avatar
SecureBio@SecureBio·Oct 16

A common argument for downplaying the threat of deliberate biological misuse is that “nature is the worst bioterrorist”. In our latest blog, we argue that nature is the baseline, not the ceiling. Optimization for harm could produce something worse than a natural pandemic. Read more: securebio.substack.com/p/nat…

Post attachment
SecureBio avatar
SecureBio@SecureBio·Oct 15

We’ve long kept our core computational pipeline public to support the wider biosurveillance field. But as we’ve scaled, the security and operational costs of this approach have grown. With this in mind, we’re shifting future bioinformatics development to private repositories. Our existing public repositories aren’t going anywhere, but will no longer be regularly updated. We’ll continue to provide high-level updates to partners who rely on our analyses and to engage with researchers and practitioners to build a more secure and sustainable biosurveillance system. Read our full blog post: securebio.org/blog/taking-ou…

SecureBio avatar
SecureBio@SecureBio·Sep 22

Frontier AI now beats the best human virologists on biosecurity-relevant evals. Once an AI model bests every human expert we can test it against, a higher score stops telling us anything about real-world risk. Read our new blog post on how bio evals have to adapt: securebio.org/blog/evaluatin…

Post attachment
AI Evaluator Forum avatar
⟳ SecureBio retweeted
AI Evaluator Forum@aievalforum·Sep 18

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: aievaluatorforum.org/initiat… Learn more at aievaluatorforum.org/path-ah…

CNBC@CNBCSep 18

Anthropic and OpenAI need truly independent safety evaluators, experts say in public letter cnbc.com/2026/09/18/ai-safet…

SecureBio avatar
SecureBio@SecureBio·Sep 18

Untargeted wastewater metagenomic sequencing screens hundreds of pathogens at once, but that breadth costs coverage, making variant tracking more difficult. However, by pooling CASPER reads nationally, we can recover real lineage and genotype trends. Read more: securebio.org/blog/tracking-…

Post attachment