←
Daniel Kang610 posts
Daniel Kang avatar

Daniel Kang

@ddkang

Distinguished AI research scientist at Bridgewater AIA labs, asst. professor at UIUC CS. Formerly in the Stanford DAWN lab and the Berkeley Sky Lab.

◷ Joined November 2010▧ 115 Photos and videos
Daniel Kang avatar
Daniel Kang@ddkang·Sep 18

We've long known that AI agent benchmarks are broken. Check out our work back from July 2025 on establishing best practices for making rigorous AI agent benchmarks! (links below)

Post attachment

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 27

When using our reward fixes and clean data, we exceed a proxy for human-level accuracy when performing self-consistency via majority voting with 16 rollouts. 5/6

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 27

In order to achieve the human-level proxy, we needed to fix two issues in the reward. First, fixing execution match issues and focusing on the domain knowledge (see the post for further details). 4/6

Post attachmentPost attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 27

By simply training on BIRD-Platinum on Kimi-K2.6, we produce a model that is well above frontier models. Our dataset also allows models to generalize to other text-to-SQL tasks. 3/6

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 27

A critical component of training models is high quality data. Prior work has shown major issues in BIRD and prior analysis from our group has both shown that: 1. Noise can be destructive to RLVR (ddkang.substack.com/p/noisy-…) 2. Major issues across a wide range of agentic benchmarks We fix the noise in BIRD, producing BIRD-Platinum as a resource for the broader community. BIRD-Platinum showcases the need for expert taste in constructing high-quality data. 2/6

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 27

New research from Bridgewater AIA Labs and UIUC, in collaboration with Thinking Machines! We use @tinkerapi to achieve state-of-the-art on a cleaned version of the BIRD text-to-SQL dataset, outperforming all frontier models we tested (including Fable and Sol) up to 8x cheaper. We achieve a proxy for human-level accuracy with no external scaffolding and a single model (w/ multiple rollouts). The common view on fine-tuning is it reduces price. Our results show the feasibility of achieving a higher performance at a fraction of the cost on economically valuable tasks, when incorporating task expertise into task-specific training. 1/6

LLMs with scaffolds have lagged on text-to-SQL, a task that relies on human judgment. By folding expert judgment into every part of RLVR on Tinker, @maxYuxuanZhu and @ddkang (UIUC and Bridgwater) trained the first text-to-SQL model to beat the human mark. thinkingmachines.ai/news/put…

Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

There is no shortcut to strong reasoning via RLVR. If we want smarter models, our empirical evidence shows that we need high-quality data. For more details, please check our paper and code: Paper: arxiv.org/abs/2603.16140 GitHub: github.com/uiuc-kang-lab/rlv… 8/9

Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

What about fancy algorithmic improvements? Can they make RLVR more robust to noisy data? Unfortunately, no. We tested SOTA algorithms: SAPO, DAPO, TIS, DR. GRPO, and PGFC (an algorithm explicitly designed to mitigate the impact of noise). When faced with a 50% error rate, all of the improved algorithms lead to > 5% accuracy degradation on at least one benchmark. 7/9

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

In addition to lower accuracy, we also find signals showing that RLVR with noisy data leads to weaker reasoning capabilities. The model trained with noisy data leads to lower pass@k (k>1) than the base model without RLVR. It also produces increasingly shorter (5–24%) outputs compared to the model trained on clean data. 6/9

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

We used Qwen2.5-Math-7B as the base model and showed that RLVR over noisy data totally collapsed. The accuracy drops by 9% compared to RLVR over clean data and is even worse than using only format-based rewards (getting rewarded for enclosing answers in \boxed{}) 5/9

Post attachment
Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

We wanted to see what happens when the data is actually noisy. Using GPT-5 Pro and human experts, we re-verified the dataset and removed 16% instances that have accidentally correct answers, building a 100% “noisy” (incorrect) dataset. Then, we ran RLVR (GRPO) again. 4/9

Daniel Kang avatar
Daniel Kang@ddkang·Aug 3

We identified a hidden flaw in prior study. We looked closely at their “100% wrong” MATH dataset and realized... It wasn't wrong. It had a massive leak of CORRECT answers hidden inside. Why? Because a MATH problem can have multiple correct answers and multiple ways to express an equivalently correct answer. However, prior data construction process completely missed them 3/9

Post attachment