←
David Van Dijcke349 posts
David Van Dijcke avatar

David Van Dijcke

@packlesshepherd

AP @UVA Econ. PhD @umichECON. Alum @lmhoxford, @LeuvenEconomics. 🇧🇪🇺🇲. Views mine.

◷ Joined April 2016▧ 77 Photos and videos
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·9 Apr 2025

I'm excited to share my job market paper (for the 2025-26 market)! It introduces a new extension of RDD where outcomes are entire distributions: Regression Discontinuity Design with Distributions (R3D). Thread below 👇 (1/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Finally, one may ask: can agents self-correct? A: no! Agents don't know what they *would* do on different data. They do try, often correcting for multiple testing. But 1) they do so inconsistently 2) they can't correct what they don't test: the garden of forking paths! (17/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

This highlights our method's value: its guarantees hold regardless of what the agents decide to do, and thus provide a durable statistical framework even as AI continues to develop rapidly. (16/)

David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Interestingly, Sonnet 5.5 is having none of it. It explicitly refuses to p-hack, and indeed: its estimated multiplier ≈ 1. No selection! Yet in our other experiments, Sonnet happily p-hacked without us even asking. (15/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Contrary to classical hypothesis testing, which assumes a hypothesis fixed in advance, the agents only write down their analysis after 1) inspecting the data, 2) estimating a few models. Often, they also revise their analysis after seeing the results. (14/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Looking at their choices, we see agents pick the specifications that deliver significant results much more frequently. (13/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

In another application, we ask the agents to estimate a difference-in-differences model to study whether minimum wages affect employment, as in Card & Krueger (1994). ... but we suggest they p-hack. The latest GPT models happily do so. Our method detects and corrects it: (12/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

In a first application, we revisit the reimplementation study of Borjas and @BreznauNate. We hand coding agents their data and the question: "does immigration reduce welfare support?", and ask the agents for the "most compelling finding" for a think tank's report. (10/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Importantly, our method is fully "black-box": no assumptions on how or even whether the agents search. If they do search, we can detect and correct it. If they don't, multiplier ≈ 1. So, do AI agents p-hack? Our results: sometimes! It depends on context and task. (9/)

David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

That gives a simple correction, building on work by Anav Sood (arxiv.org/abs/2411.13764): Our theoretical results show that the corrected p is asymptotically valid and consistent. In words: our method corrects any agent-driven distortions to statistical validity! (8/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

The idea: replay the same agent, with the same prompt, on subsamples of the data and log which analysis it reports each time. Then count: how often does it reselect its *original* analysis when that looked good statistically (smaller p) vs. not? --> "selection multiplier" (7/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

We formalize coding agents engaging in a new type of selective inference, combining human-like search over a vast analysis space with programmatic replayability of that search. This suggests a new way to correct selection that wasn't possible with humans: just replay them! (6/)

David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Our simulations back this up. When given a standard analytical task and asked to produce "one compelling, defensible finding", the latest frontier agents (GPT-6--6.2, Claude 5.5) report significant results up to 98% of the time on data **with no effects** (5/)

Post attachment
David Van Dijcke avatar
David Van Dijcke@packlesshepherd·5h

Empirical analyses that used to take weeks can now be run in minutes! Even without explicit instructions to hunt for results, coding agents will often inspect the data they are given before deciding on a null hypothesis and statistical model, making those data-dependent (4/)