
Paper: arxiv.org/abs/2611.07461 Blog: dream.ischool.illinois.edu/b… Platform: heelo.dev

Assistant professor @iSchoolUI at UIUC, affiliated at CS and IGB. Previously @CarnegieMellon (across CBD, LTI, MLD). Trustworthy AI. & Computational Biology

Paper: arxiv.org/abs/2611.07461 Blog: dream.ischool.illinois.edu/b… Platform: heelo.dev

Caveat worth stating plainly: the 46 live tasks were written by the same team that built the platform, criteria fixed before the first run but primitives fixed and re-run when a task exposed a defect. An agent on its own had no credentials. Preprint, not peer-reviewed.

The control I find most convincing: spend the same calls and the same backbone on self-refining arguments over raw tool schemas instead, and you get 34.2% on ToolBench against 75.1%. Showing the caller the same code components as read-only context gets 0.3544 delivery against 0.5134. It isn’t more inference, and it isn’t more access. It’s where the applying happens.

Eight families, twenty-plus benchmarks, each against its own study’s baseline: ToolBench 55.8% → 75.1% Terminal-Bench 4.0 37.3% (Codex) → 55.5%, same backbone GAIA-Text-103 45.6% → 58.3% Putnam hit@5 84.2% → 95.0% Repository delivery 0.3180 → 0.5134

The work from @Haibo_Jin97, @Krumper2312, Xiaopeng Yuan and the team: let the competence travel with the resource. An agent-native primitive = a reusable resource + a resident LLM that applies it in context and messages the other primitives its result constrains. HEELO then composes the few primitives a task needs into a graph, verifies against the task’s own evidence, and revises only what the failure implicates.

Why do AI agents fail at real-world tasks? Not at reaching a resource — that problem is basically solved by tool catalogues, retrieval and MCP. They fail at the last mile: applying it. A call has to ground every argument in an earlier call’s output. A retrieved component has to be adapted to its target. A proof method only applies where its conditions hold. Today all of that sits with the caller, so every harness rebuilds it for every task.

Claude Code completed 9 of 46 real-world tasks on its own. Connected to a library of agent-native primitives through MCP, the same agent completed 39. Codex went from 9 to 35. The model didn’t change. What changed is who applies the tool.

Connect your agent through MCP or the API, and enjoy the magic: heelo.dev

HEELO builds each task's harness from a library of primitives, checks the result against real evidence, and revises the primitives a failure points to. huggingface.co/papers/2611.0… If the paper is useful, an upvote on HF helps 🙏

After half a year of baking, we are releasing our grand paper: The Last Mile to the Real World. Agents can reach tools but often can't apply them. We give each resource a resident LLM: an agent-native primitive. With HEELO, Claude Code goes from 9 to 39 of 46 real-world tasks, Codex from 9 to 35. huggingface.co/papers/2611.0…

Blog: dream.ischool.illinois.edu/b… Led by @kevvvv123123 , with Ye Yu, John Sciarappo, Haojing Luo, @kaiminchang, @yichiaw

Our planner-writer method, ThreadPDG, passed 58–100% with two LLMs, but only 17–42% with gpt-4o-mini. It needs a real seed thread and inherits its reply tree, so it’s a case study, not a solved problem. Politeness and disagreement are still off.

Here’s the uncomfortable part. Human raters didn’t pick out the real thread in about 60% of pairs for those same simulators. If your validation is “it looks right to me”, you are not validating much.

Two existing simulators, OASIS and SynthPAI, passed 0–17% of the metrics in every domain, with all three LLMs. With gpt-5.4-mini, OASIS never replied to another comment (0% vs 46% in real threads), and 35% of its comments opened with agreement, against 6% in real ones.

So we built ThreadBench (first posted as MiroBench): 9,817 real Reddit threads, 8 domains, 12 thread-level metrics covering repetition, expression, tone and reply structure. Two samples of real threads pass each other 94.30% of the time. The bar is one that real discussions clear.

Fluency is the wrong test. Real threads are uneven: people push back, tell stories, ask questions, and leave five words and go. A simulated crowd can sound human one comment at a time and still not behave like a crowd.