←
Drew Prinster286 posts
Drew Prinster avatar

Drew Prinster

@DrewPrinster

Safe & regulatable AI/ML for health | My job is (mostly) error bars 🫡 (eg, conformal prediction) | CS PhD at Johns Hopkins. Prev at Yale. he/him

◷ Joined December 2019▧ 56 Photos and videos
Neha Verma avatar
⟳ Drew Prinster retweeted
Neha Verma@n_verma1·Nov 4

Need different model weights? Just change them 🙃 Our new paper VFold does exactly that to enable low-overhead cross-layer value cache compression: half the value cache, >98% of full-cache performance 🧵 📄 arxiv.org/abs/2610.12338

Post attachment
JHU CLSP avatar
⟳ Drew Prinster retweeted
JHU CLSP@jhuclsp·Oct 28

✨ Interested in applying to Johns Hopkins’ Computer Science Ph.D. program? ✨ JHU CLSP is offering a student-run application mentoring program for those seeking additional feedback on their application materials. ...

Kyunghyun Cho avatar
⟳ Drew Prinster retweeted
Kyunghyun Cho@kchonyc·Oct 26

Ortet's mission is to help create healthcare where everyone can receive predictably better care. in pursuing this mission, we believe predictability must come together with transparency and controllability. taking the first step in this direction of transparency and predictability in health, we have created the Orion cluster, host a large fleet of open-weight models and began our evalution campaign on health-focused benchmarks. in doing so, we learned a lot about building up an agent-first cluster and also learned about the current state of AI models in the context of health benchmarks. we are excited to share the first set of findings from this ongoing campaign on our Orion cluster.

Ortet@OrtetAIOct 26

We’re releasing Ortet’s first health evaluation results: 34 open-weight models, 50 tasks, and 15 public benchmarks served and evaluated on Orion, our NVIDIA Blackwell GPU cluster. A few findings: - General rankings are not enough for health. - Open models are competitive on basic medical knowledge and are more willing research collaborators. - Both open and closed models struggle on the patient-prediction tasks we tested. - More reasoning usually helps, but the highest setting doesn’t always produce the best observed score. - Health rankings can be estimated with a small subset of questions. This is the first step towards our goal—to build continuously improving intelligence across the clinical, biological, and operational aspects of health. Read the blogpost: ortet.ai/blog/open-models-in… More on the findings below 🧵

Drew Nguyen avatar
⟳ Drew Prinster retweeted
Drew Nguyen@drewtnguyen·Oct 13

# A statistical look back at METR’s time horizons—and a look ahead Here are the 228 software tasks from METR’s time horizons chart, with human completion time (x) against estimated difficulty for AI (y). Does time proxy difficulty well? With @wfithian, I quantified this. 🧵

Post attachment
Shuvom Sadhuka avatar
⟳ Drew Prinster retweeted
Shuvom Sadhuka@shuvom_s·Oct 4

An underrated fact I recently learned: any ensemble of calibrated predictors will necessarily be uncalibrated (unless all the predictors are copies of each other). How bad is the gap in practice? I ensembled some calibrated reward models. Even though ensembling improves the AUC (compared to each individual RM), it dramatically worsens the ECE. Blog post below:

Post attachment
jenny huang avatar
⟳ Drew Prinster retweeted
jenny huang@jennyihuang·Sep 30

new preprint: language models are “insecure” reporters. researchers and engineers have become increasingly reliant on LLM-written reports---second-hand accounts of code and experiments---to understand agent-generated work. in this study, we ask: “how transparent are these LLM-generated reports?” 🔍

Post attachment
Drew Prinster avatar
Drew Prinster@DrewPrinster·Sep 29

No one paper, team, lab, etc is going to solve AI safety alone. But regulating how much a new frontier model version can diverge from a safe predecessor version seems like a desirable puzzle piece. Some thoughts on this distilled in links below

Drew Prinster avatar
Drew Prinster@DrewPrinster·Sep 29

but as the daily news suggests, it does seem like “guaranteeing that next-gen Mythos/Astra/etc will satisfy our declared safety constraints, given data from current gen models” would be nice. Lots needs to be done here....

Exclusive: OpenAI is scrapping the release of its next-generation AI model because it failed to meet safety standards on.wsj.com/4xPPgao

Drew Prinster avatar
Drew Prinster@DrewPrinster·Sep 29

The question of how to “pace the frontier," or regulate a “speed limit” on how fast we can advance AI capabilities while ensuring safety, sounds a bit like a “safe policy improvement” problem to me. Obviously I’m biased and there’s many social, political, etc, factors at play...

Can we ensure AI agents respect our safety constraints, even as they explore & improve? - Medical LLMs that are helpful, & avoid false claims? - Bioscience agents that generate effective molecule designs, & ensure they’re safe? 📄🧵w/ @samuel_stanton_ @clara_fannjiang @jiwoncpark @kchonyc @anqi_liu33 @suchisaria Excited to share “Conformal Policy Control” ⬇️ 1/12

Kyunghyun Cho avatar
⟳ Drew Prinster retweeted
Kyunghyun Cho@kchonyc·Sep 29

i was diagnosed with thyroid cancer many years ago purely by chance, because i had to get blood work for my US green card application (thank you, USCIS!). i was in my mid-30’s and was reasonably healthy (or so i believed.) this was shocking and totally unpredictable news to me. thanks to the amazing team of clinicians at NYU Langone, i successfully went through a total thyroidectomy followed by radioactive iodine therapy. but, it was truly the most uncertain and confusing period of my life. this is when i first realized that no one can opt out of healthcare. being sick isn’t what you choose to or choose not to. it is something that unpredictably happens to all of us at some point for one reason or another. this is why it is a healthcare system that we collectively build and use to support each other at the society level. because it must support everyone, the complexity of creating, maintaining and improving this system is a Herculean effort that calls for all the help in the world. today, i am announcing @OrtetAI together with my dear co-founders, @keunwoochoi, @HenriDwyer, @elmanmansimov, @causalclaudia and Jeff, to lend our support to improving the health of each and every one of us. Ortet’s goal is to create a technological foundation that will enable health systems to deliver predictably better care to each and every patient. this will only be possible by working closely with all stakeholders in the broader ecosystem, which is enabled in part by Ortet’s close partnership with Thoreau. Ortet is taking the very first step of a long journey ahead, and at the end of this journey, i strongly believe we would be able to say that Ortet had contributed to building better, more predictable healthcare and thus to improving health of each and every one of us. see ortet.ai/ for more!

Drew Prinster avatar
Drew Prinster@DrewPrinster·Sep 29

It's not all rainbows of course, every PhD experience is different, and there's a lot of luck and factors out of your control involved. And, there's plenty of good reasons not to do a PhD. But Phillip's thoughts resonated w me. If you do it, make sure to imagine Sisyphus happy.