
Paper: arxiv.org/abs/2507.02825 Blog: ddkang.substack.com/p/ai-age…

Distinguished AI research scientist at Bridgewater AIA labs, asst. professor at UIUC CS. Formerly in the Stanford DAWN lab and the Berkeley Sky Lab.

Paper: arxiv.org/abs/2507.02825 Blog: ddkang.substack.com/p/ai-age…

We've long known that AI agent benchmarks are broken. Check out our work back from July 2025 on establishing best practices for making rigorous AI agent benchmarks! (links below)
Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

And see our technical report for more details: arxiv.org/abs/2603.20004

This work is joint w/ @maxYuxuanZhu , @tengjun_77 , and @yoojinc3, in collaboration with Thinking Machines. 6/6

When using our reward fixes and clean data, we exceed a proxy for human-level accuracy when performing self-consistency via majority voting with 16 rollouts. 5/6

In order to achieve the human-level proxy, we needed to fix two issues in the reward. First, fixing execution match issues and focusing on the domain knowledge (see the post for further details). 4/6

By simply training on BIRD-Platinum on Kimi-K2.6, we produce a model that is well above frontier models. Our dataset also allows models to generalize to other text-to-SQL tasks. 3/6

A critical component of training models is high quality data. Prior work has shown major issues in BIRD and prior analysis from our group has both shown that: 1. Noise can be destructive to RLVR (ddkang.substack.com/p/noisy-…) 2. Major issues across a wide range of agentic benchmarks We fix the noise in BIRD, producing BIRD-Platinum as a resource for the broader community. BIRD-Platinum showcases the need for expert taste in constructing high-quality data. 2/6

New research from Bridgewater AIA Labs and UIUC, in collaboration with Thinking Machines! We use @tinkerapi to achieve state-of-the-art on a cleaned version of the BIRD text-to-SQL dataset, outperforming all frontier models we tested (including Fable and Sol) up to 8x cheaper. We achieve a proxy for human-level accuracy with no external scaffolding and a single model (w/ multiple rollouts). The common view on fine-tuning is it reduces price. Our results show the feasibility of achieving a higher performance at a fraction of the cost on economically valuable tasks, when incorporating task expertise into task-specific training. 1/6
LLMs with scaffolds have lagged on text-to-SQL, a task that relies on human judgment. By folding expert judgment into every part of RLVR on Tinker, @maxYuxuanZhu and @ddkang (UIUC and Bridgwater) trained the first text-to-SQL model to beat the human mark. thinkingmachines.ai/news/put…

Co-written with @maxYuxuanZhu 9/9

There is no shortcut to strong reasoning via RLVR. If we want smarter models, our empirical evidence shows that we need high-quality data. For more details, please check our paper and code: Paper: arxiv.org/abs/2603.16140 GitHub: github.com/uiuc-kang-lab/rlv… 8/9

What about fancy algorithmic improvements? Can they make RLVR more robust to noisy data? Unfortunately, no. We tested SOTA algorithms: SAPO, DAPO, TIS, DR. GRPO, and PGFC (an algorithm explicitly designed to mitigate the impact of noise). When faced with a 50% error rate, all of the improved algorithms lead to > 5% accuracy degradation on at least one benchmark. 7/9

In addition to lower accuracy, we also find signals showing that RLVR with noisy data leads to weaker reasoning capabilities. The model trained with noisy data leads to lower pass@k (k>1) than the base model without RLVR. It also produces increasingly shorter (5–24%) outputs compared to the model trained on clean data. 6/9

We used Qwen2.5-Math-7B as the base model and showed that RLVR over noisy data totally collapsed. The accuracy drops by 9% compared to RLVR over clean data and is even worse than using only format-based rewards (getting rewarded for enclosing answers in \boxed{}) 5/9

We wanted to see what happens when the data is actually noisy. Using GPT-5 Pro and human experts, we re-verified the dataset and removed 16% instances that have accidentally correct answers, building a 100% “noisy” (incorrect) dataset. Then, we ran RLVR (GRPO) again. 4/9

We identified a hidden flaw in prior study. We looked closely at their “100% wrong” MATH dataset and realized... It wasn't wrong. It had a massive leak of CORRECT answers hidden inside. Why? Because a MATH problem can have multiple correct answers and multiple ways to express an equivalently correct answer. However, prior data construction process completely missed them 3/9