I believe one of most impactful (and easiest) contributions frontier AI labs could make to science would be to run AI reviews of entire the open-access literature, including supplements, and make the reviews freely available to everyone.
PMC has ~8.4M open-access articles. For this rough estimate, I used 3M papers as a starting point, focusing on original research and screening out low-quality stuff etc.
Estimated model cost: $0.6–6M, depending on the model. Then keep it running for every new bioRxiv/medRxiv preprint. I believe this is a super small amount considering what we already spend on research. The US alone spent an estimated $993B on R&D in 2024. Even the upper estimate of $6M is less than 0.001% of that.
+ hopefully in the long run, preprints + public AI and human review will eventually replace journals.
New blog post: I used Claude to audit the reproducibility of every journal paper I've been an author on in the last five years. Our commitment to openness meant that the code and data were available, but there were some significant issues. 🧵babieslearninglanguage.blogs…
Completely feel the gap increasing between AI-fueled progress at the frontier and everything else.
An interesting byproduct of sitting on the steep part of the curve is that you can no longer even rely on your previously held well-considered beliefs about …well anything.
I frequently find I can’t confidently say what I think about the state of science and its future and how it should be organized or how we should operate because I’m constantly like .. wait is that still what I think? or has 10 minutes gone by and changed the parameters and calculus. It leads to a kind of mental-updating fatigue for sure.
Having spent a considerable amount of my professional life trying to get researchers to stop offboarding their scientific judgement to journal, grant, and hiring panels, it feels like there’s now a completely new threat in the age of AI…offboarding your judgment to your own rapidly irrelevant prior opinion.
It’s almost incomprehensible how wide the distance is between someone who’s working at the frontier with AI and basically everyone else right now.
The crazy thing is these lines are not converging, but only growing farther apart. If the curve of AI progress had leveled off, it would certainly have given most people and companies time to catch up or implement their workflows in a standardized way. The technology would have neutralized competitive advantage because everyone would be using it the same way.
But what’s happened is the opposite. The curve of AI progress is basically going vertical right now, which means if you had codified your processes 3-6 months ago you’re likely already behind again. Now imagine most companies that kicked into gear a year ago and thought that they were done. Even further behind now.
I’ve never seen a technology in my life and career that improves at this rate, which consequently means you need to upgrade your expectations at a similar rate. Our brains weren’t wired for this, and our organizational change management processes certainly weren’t either.
With any form of recursive self improvement, things just get wilder. It’s a time when you will need to be constantly reevaluating the things that didn’t work just a couple months ago to see if they now can be solved. AI native companies certainly have a strong advantage due to this right now, so now it comes down to can the bigger companies keep up. Wild times.
We have to stop using "paper count" as a useful scientific metric. More writing output no longer means more knowledge gained
washingtonpost.com/education…
Today I'm posting a new blog on a familiar topic... Nearly two hundred papers have confused the names of different proteins and recorded using the wrong antibody! This time it's PERK vs. p-ERK. Researchers from Harvard, the Mayo Clinic and more. Link in the first comment.
The Metacheck team is busy building an improved version of the osf_download function (scienceverse.org/metacheck_b… ). You will soon be able to use metacheck to download all files, wikis, and meta-data. You will also be able to upload everything to Zenodo.
Big news! 🎉
BrRN has just received a grant from @TempletonWorld for a research project evaluating strategies to promote open and reproducible practices in Brazilian graduate programs.
reprodutibilidade.org/en/pos…
Grande novidade! 🎉
A RBR acaba de receber um grant da @TempletonWorld para um projeto de pesquisa que avalia estratégias de promoção de práticas abertas e reprodutíveis nos programas de pós-graduação brasileiros.
reprodutibilidade.org/post/r…
BRISA, our Brazilian preclinical systematic review collaboration, has a new preprint on the effect of NMDA antagonists on social behaviour. Check it out at biorxiv.org/content/10.64898…
We are pleased to announce the launch of 10 recommendations for open and reproducible research in graduate programs!
The document presents practical and adaptable guidelines, including resources, tools, and examples.
Read more (in Portuguese) here: osf.io/m4csj
🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science.
Here’s what we found 👇
Join us on Thursday, March 5, for a free webinar exploring what Diamond Open Access is, why it matters for fair and equitable scholarly publishing and how to transition to community-led, author- and reader-free publishing models.
elifesciences.org/events/942…
eLife’s Global South Committee is hosting a free webinar exploring Diamond Open Access, featuring a range of industry perspectives and highlighting successful implementations from around the world.
Register free: elifesciences.org/events/942…