
🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science. Here’s what we found 👇

The Institute for Replication (I4R) works to improve the credibility of science by promoting and conducting reproductions and replications.

🧵1/ Our first meta-science paper (with 350+ coauthors) is published today in Nature. It presents one of the largest-ever reproducibility projects in economics & political science. Here’s what we found 👇

We reproduced 125 articles from a couple of sports/exercise journals over the past two weeks. Emailing all reports to authors this week. We will be opening registration for a similar virtual forensic games focusing on physiotherapy tomorrow. Stay tuned!

#R2EDev #3: Chess instruction reduces children's risk aversion in Bangladesh. Originally published in the JDE, the paper is robust. Study was implemented by #GDRI_rep & the replication reveals unusual data patterns. Authors did not respond @I4Replication econstor.eu/bitstream/10419/…

#R2EDev #2: Findings on gambling & liquidity needs in Uganda, originally published in AEJ, are largely robust to alternative analytical choices. Author did not respond. @I4Replication econstor.eu/bitstream/10419/…

#R2EDev #1: Findings on intergenerational entrepreneurship, originally published in the JDE, are largely not robust to modest changes, such as controlling for parental education or including informal business owners in outcome definition. @I4Replication econstor.eu/bitstream/10419/…

Tomorrow (Fri 13 Nov): Webinar on open science, robust transparent research, open evaluation & replication, & widening India/LMIC participation. unjournal.org @monkprayogshala @I4Replication #openscience #OpenAccess #replication #econtwitter #psychtwitter

How robust is development economics? 66 randomly selected papers from the AER, AEJ:AE & JDE. 66 (human!) replicators. Pre-specified robustness checks – mostly conservative ones. Replication reports will now appear on a rolling basis. @I4Replication #R2EDev i4replication.org/blog/repro…

Toulouse Dec 8th: surveymonkey.ca/r/5BMYXMD Poitiers Dec 10th: surveymonkey.ca/r/562W5SX Montpellier Dec 10th: surveymonkey.ca/r/562XW3K

We are hosting 3 Replication Games in France on December 8th and 10th, 2026. Cities: Toulouse, Poitiers and Montpellier. Social science articles will be reproduced! Registration links on the posters and below.

Forensic metascience is growing fast, but it's scattered. This question came out of a conversation with the founder of @MetascienceObs: what if we worked like one field, with a clear division of labor? Numbers: the Forensic Metascience AI Toolkit from @MetascienceObs runs GRIM, SPRITE, statcheck and Table 1 checks on a paper's own figures. Images: @MicrobiomDigest and the image sleuths. Replication: @I4Replication reruns published analyses. Retractions: @RetractionWatch keeps the record. Citations: SaneCite checks what a paper cites. Across 51,626 citations to retracted papers, AI readings found 40.7% still treat the retracted work as support (pending human review). One banner, shared lists, no duplicated work. Who's missing from this map? Check any paper's references free: sanecite.com/checker#paper

We are hosting Replication Games on January 21, 2027 at University of Cologne. Economic articles will be reproduced! Register here: surveymonkey.ca/r/cologne_re…

The people most vocal about errors in econ papers (there are many!) are motivated, and trying to take down specific people (e.g. Zucman), or debunk the field as a whole. If we aren't willing to criticize ourselves / each other, we do these people a favor by making them look like the only truth seekers. Tons of econ papers have errors, and those errors will come to light. This moment is a test of our belief that econ is the most truth-seeking of the social sciences. Trying to bury and obfuscate the errors will hurt us in the long run. Some papers will need retractions, all will need errata. It's an individual nuisance—I'm busy too and don't want to be bothered to go and re-replicate and audit all my past work. But if we don't show a willingness to cleaning up our own house, we can count on our opponents to do it. Whether we will be more honest about our errors than our opponents remains to be seen. I suspect the risks of admitting you were wrong are probably lower than you think. It will take some brave authors to lead the way and set the standard.

Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published. Some open questions: 1. How good is today’s AI for this? 2. How much would it cost? 3. Do you get most of the value with a targeted subset of research? 4. Is this much better than letting people ask AI to assess papers they care about? 5. What to do with papers whose data isn’t available? 6. What kind of organization would be best to lead the project? 7. What do you do with the outputs of the project? My (preliminary!) thoughts on the above: 1. How good is today’s AI for this? I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it. 2. How much would it cost? ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide. 3. Do you get most of the value with a targeted subset of research? Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years? 4. Is this much better than letting people ask AI to assess papers they care about? I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say. 5. What to do with papers whose data isn’t available? In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort? 6. What kind of organization would be best to lead the project? A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization? 7. What do you do with the outputs of the project? Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?


We are hosting Replication Games on Friday November 27th, 2026 at Uni Marburg. Studies on conflict, political violence, and early childhood interventions for education and health will be reproduced! Register here: surveymonkey.ca/r/5Z3L7V8

Serious data sharing policy and looking for a data editor. That's amazing and quite impressive given the norms in this field. @JSportsSci