←
Haohan Wang418 posts
Haohan Wang avatar

Haohan Wang

@HaohanWang

Assistant professor @iSchoolUI at UIUC, affiliated at CS and IGB. Previously @CarnegieMellon (across CBD, LTI, MLD). Trustworthy AI. & Computational Biology

◷ Joined November 2012▧ 89 Photos and videos
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

Caveat worth stating plainly: the 46 live tasks were written by the same team that built the platform, criteria fixed before the first run but primitives fixed and re-run when a task exposed a defect. An agent on its own had no credentials. Preprint, not peer-reviewed.

Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

The control I find most convincing: spend the same calls and the same backbone on self-refining arguments over raw tool schemas instead, and you get 34.2% on ToolBench against 75.1%. Showing the caller the same code components as read-only context gets 0.3544 delivery against 0.5134. It isn’t more inference, and it isn’t more access. It’s where the applying happens.

Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

Eight families, twenty-plus benchmarks, each against its own study’s baseline: ToolBench 55.8% → 75.1% Terminal-Bench 4.0 37.3% (Codex) → 55.5%, same backbone GAIA-Text-103 45.6% → 58.3% Putnam hit@5 84.2% → 95.0% Repository delivery 0.3180 → 0.5134

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

The work from @Haibo_Jin97, @Krumper2312, Xiaopeng Yuan and the team: let the competence travel with the resource. An agent-native primitive = a reusable resource + a resident LLM that applies it in context and messages the other primitives its result constrains. HEELO then composes the few primitives a task needs into a graph, verifies against the task’s own evidence, and revises only what the failure implicates.

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

Why do AI agents fail at real-world tasks? Not at reaching a resource — that problem is basically solved by tool catalogues, retrieval and MCP. They fail at the last mile: applying it. A call has to ground every argument in an earlier call’s output. A retrieved component has to be adapted to its target. A proof method only applies where its conditions hold. Today all of that sits with the caller, so every harness rebuilds it for every task.

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 9

Claude Code completed 9 of 46 real-world tasks on its own. Connected to a library of agent-native primitives through MCP, the same agent completed 39. Codex went from 9 to 35. The model didn’t change. What changed is who applies the tool.

0:47
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 6

HEELO builds each task's harness from a library of primitives, checks the result against real evidence, and revises the primitives a failure points to. huggingface.co/papers/2611.0… If the paper is useful, an upvote on HF helps 🙏

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 6

After half a year of baking, we are releasing our grand paper: The Last Mile to the Real World. Agents can reach tools but often can't apply them. We give each resource a resident LLM: an agent-native primitive. With HEELO, Claude Code goes from 9 to 39 of 46 real-world tasks, Codex from 9 to 35. huggingface.co/papers/2611.0…

0:21
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 5

Our planner-writer method, ThreadPDG, passed 58–100% with two LLMs, but only 17–42% with gpt-4o-mini. It needs a real seed thread and inherits its reply tree, so it’s a case study, not a solved problem. Politeness and disagreement are still off.

Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 5

Here’s the uncomfortable part. Human raters didn’t pick out the real thread in about 60% of pairs for those same simulators. If your validation is “it looks right to me”, you are not validating much.

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 5

Two existing simulators, OASIS and SynthPAI, passed 0–17% of the metrics in every domain, with all three LLMs. With gpt-5.4-mini, OASIS never replied to another comment (0% vs 46% in real threads), and 35% of its comments opened with agreement, against 6% in real ones.

Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 5

So we built ThreadBench (first posted as MiroBench): 9,817 real Reddit threads, 8 domains, 12 thread-level metrics covering repetition, expression, tone and reply structure. Two samples of real threads pass each other 94.30% of the time. The bar is one that real discussions clear.

Post attachment
Haohan Wang avatar
Haohan Wang@HaohanWang·Nov 5

Fluency is the wrong test. Real threads are uneven: people push back, tell stories, ask questions, and leave five words and go. A simulated crowd can sound human one comment at a time and still not behave like a crowd.

Post attachment