←
Mathew Koretsky22 posts
Mathew Koretsky avatar

Mathew Koretsky

@mkoretsky1

machine learning engineer | @DataTecnica | @NIH @BiomedArena | @uvmvermont | views/tweets are my own

◷ Joined December 2022
Faraz Faghri avatar
⟳ Mathew Koretsky retweeted
Faraz Faghri@FarazFaghri·Jan 12

🚀 We just launched DTAgent-AD, a scientific reasoning agent built to accelerate biomedical research and specialized Alzheimer’s & neurodegenerative disease research. DTAgent-AD is a domain-tuned agent trained on trace data from real biomedical tasks, not a general chatbot.

Post attachment
Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·22 Dec 2025

Despite boasting impressive performance across a range of categories, the latest frontier LLMs (Gemini 3 Pro, Claude Opus 4.5, and GPT-5.2) still struggle to balance accuracy and safety on CARDBiomedBench, our biomedical QA benchmark 👀

Frontier models are moving fast, but are they getting better at biomedical research? We just ran a fresh benchmark update using CARDBiomedBench, our evaluation suite for genetics, disease associations, and drug discovery QA. Instead of looking only at “did it answer?”

Post attachment
Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·4 Nov 2025

👀 Our new Knowledge Agents make BiomedArena.AI more powerful than ever. 🧬 Get insights backed by the journals and databases that biomedical researchers use daily. 📄 Read the blog more more info: datatecnica.com/blog/-biomed…

🧬 New at BiomedArena.AI: smarter Biomedical Knowledge Agents + Knowledge Mode We just shipped the latest update to BiomedArena.AI, the world’s first platform for benchmarking LLMs on biomedical research tasks.

Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·7 Oct 2025

We continue to evaluate these new models on our benchmark, CARDBiomedBench. Despite significant progress, there are still no models that balance response accuracy and safety on biomedical questions 👀

We evaluated 12 top models using CARDBiomedBench, a biomedical benchmark with 68K+ expert QA pairs across GWAS, SMR, drug discovery & more. 🧠 No model aced both safety and accuracy. 🤖 GPT-4o = bold but risky 🤔 Claude-4.0 = cautious but wrong More is coming soon.

Post attachment
Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·7 Oct 2025

Check out the latest models in @BiomedArena for all of your biomedical research questions!

🚀 New LLMs now LIVE on BiomedArena 🧬 Test GPT-5, Claude-4.1, Gemini 2.5 and more, on your toughest biomedical queries. All free. All benchmarked. biomedarena.ai 📉 Can AI be accurate and safe in biomedicine? See the surprising results 👇🧵

Post attachment
Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·19 Aug 2025

🚨BiomedArena is live🚨 In a partnership with @lmarena_ai, our team at @DataTecnica has released a feedback-rich platform to evaluate LLM performance on real-world biomedical questions. ⚔️Access the arena: biomedarena.ai 📄Read the blog post: news.lmarena.ai/introducing-…

Arena.ai@arena19 Aug 2025

🧬 BiomedArena is here! We’re honored to partner with @DataTecnica and @NIH CARD, who developed BiomedArena to evaluate LLMs for biomedical discovery, and to help expand this domain-specific track in community-driven evaluations. 🧪 Biomedical science is complex, high-stakes, and constantly evolving. 📊 CARDBiomedBench & tabular reasoning tests show that no current model can reliably meet the reasoning & domain-specific knowledge demands of biomedical researchers. Learn more about BiomedArena in thread 👇 🧵 #AI #LLMs #BiomedicalAI #AIEvaluation #OpenScience #LMArena #BiomedArena #NIH

Post attachment
Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·28 May 2025

We believe this benchmark is a critical step towards building trustworthy text-to-SQL systems that can increase efficiency of lookups for PIs and SMEs, democratize access to biomedical knowledge, and accelerate discovery

Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·28 May 2025

Top performers: 1) GPT-o3-mini is the top frontier model with an accuracy of 59% on BiomedSQL, significantly trailing expert-level performance (90%) 2) Our custom text-to-SQL system, BMSQL, improves execution accuracy by ~3% and response quality by ~7% over “vanilla” GPT-o3-mini

Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·28 May 2025

Biomedical researchers increasingly rely on large-scale databases that store electronic health records, population-scale studies, and clinical trial information. Our team wanted to test LLMs ability to generate valid SQL queries from questions that domain experts routinely ask

Mathew Koretsky avatar
Mathew Koretsky@mkoretsky1·28 May 2025

Can LLMs perform reliably as biomedical data analysts? TL;DR: We created the first benchmark designed to challenge LLMs ability to apply scientific reasoning in text-to-SQL generation over biomedical databases, revealing a 30-40% gap between SOTA models and expert performance