←
I4R2,491 posts
I4R avatar

I4R

@I4Replication

The Institute for Replication (I4R) works to improve the credibility of science by promoting and conducting reproductions and replications.

◷ Joined December 2021▧ 398 Photos and videos
I4R avatar
I4R@I4Replication·Apr 1

🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science. Here’s what we found 👇

Post attachment
I4R avatar
I4R@I4Replication·3h

We reproduced 125 articles from a couple of sports/exercise journals over the past two weeks. Emailing all reports to authors this week. We will be opening registration for a similar virtual forensic games focusing on physiotherapy tomorrow. Stay tuned!

𝙳𝚊𝚟𝚒𝚍 𝚁𝚎𝚒𝚗𝚜𝚝𝚎𝚒𝚗 avatar
⟳ I4R retweeted
𝙳𝚊𝚟𝚒𝚍 𝚁𝚎𝚒𝚗𝚜𝚝𝚎𝚒𝚗@GivingTools·Nov 12

Tomorrow (Fri 13 Nov): Webinar on open science, robust transparent research, open evaluation & replication, & widening India/LMIC participation. unjournal.org @monkprayogshala @I4Replication #openscience #OpenAccess #replication #econtwitter #psychtwitter

I4R avatar
I4R@I4Replication·Nov 5

We are hosting 3 Replication Games in France on December 8th and 10th, 2026. Cities: Toulouse, Poitiers and Montpellier. Social science articles will be reproduced! Registration links on the posters and below.

Post attachmentPost attachmentPost attachment
SaneCite avatar
⟳ I4R retweeted
SaneCite@SaneCite·Nov 3

Forensic metascience is growing fast, but it's scattered. This question came out of a conversation with the founder of @MetascienceObs: what if we worked like one field, with a clear division of labor? Numbers: the Forensic Metascience AI Toolkit from @MetascienceObs runs GRIM, SPRITE, statcheck and Table 1 checks on a paper's own figures. Images: @MicrobiomDigest and the image sleuths. Replication: @I4Replication reruns published analyses. Retractions: @RetractionWatch keeps the record. Citations: SaneCite checks what a paper cites. Across 51,626 citations to retracted papers, AI readings found 40.7% still treat the retracted work as support (pending human review). One banner, shared lists, no duplicated work. Who's missing from this map? Check any paper's references free: sanecite.com/checker#paper

Paul Novosad avatar
⟳ I4R retweeted
Paul Novosad@paulnovosad·Oct 29

The people most vocal about errors in econ papers (there are many!) are motivated, and trying to take down specific people (e.g. Zucman), or debunk the field as a whole. If we aren't willing to criticize ourselves / each other, we do these people a favor by making them look like the only truth seekers. Tons of econ papers have errors, and those errors will come to light. This moment is a test of our belief that econ is the most truth-seeking of the social sciences. Trying to bury and obfuscate the errors will hurt us in the long run. Some papers will need retractions, all will need errata. It's an individual nuisance—I'm busy too and don't want to be bothered to go and re-replicate and audit all my past work. But if we don't show a willingness to cleaning up our own house, we can count on our opponents to do it. Whether we will be more honest about our errors than our opponents remains to be seen. I suspect the risks of admitting you were wrong are probably lower than you think. It will take some brave authors to lead the way and set the standard.

Matt Clancy avatar
⟳ I4R retweeted
Matt Clancy@mattsclancy·Oct 29

Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published. Some open questions: 1. How good is today’s AI for this? 2. How much would it cost? 3. Do you get most of the value with a targeted subset of research? 4. Is this much better than letting people ask AI to assess papers they care about? 5. What to do with papers whose data isn’t available? 6. What kind of organization would be best to lead the project? 7. What do you do with the outputs of the project? My (preliminary!) thoughts on the above: 1. How good is today’s AI for this? I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it. 2. How much would it cost? ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide. 3. Do you get most of the value with a targeted subset of research? Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years? 4. Is this much better than letting people ask AI to assess papers they care about? I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say. 5. What to do with papers whose data isn’t available? In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort? 6. What kind of organization would be best to lead the project? A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization? 7. What do you do with the outputs of the project? Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?

I4R avatar
I4R@I4Replication·Oct 20

Already 115 registered participants within 24 hours. 10 slots left!

Post attachment
I4R avatar
I4R@I4Replication·Oct 20

We are hosting Replication Games on Friday November 27th, 2026 at Uni Marburg. Studies on conflict, political violence, and early childhood interventions for education and health will be reproduced! Register here: surveymonkey.ca/r/5Z3L7V8

Post attachment
I4R avatar
I4R@I4Replication·Oct 19

Serious data sharing policy and looking for a data editor. That's amazing and quite impressive given the norms in this field. @JSportsSci

Post attachmentPost attachment