
The best frontier model (Opus 5.5) solves 18.5% of long-horizon coding tasks, even with a clear correctness oracle. With four attempts it solves 31.9%. 38 of 69 tasks were solved by no model. Setup - Multi-step SWE: 69 tasks, each a chain of 5–15 dependent, multi-hour steps on one codebase, each building on the agent’s own earlier code - Four kinds of work: optimization, migration, feature implementation, and version upgrades (hardest: 2.9% solved across models) - Each step’s instruction and grader derive from the same outside source of truth (so they can’t drift apart) - Two rounds of expert review per task, graders rebuild from source and run unprivileged, 56 of 69 tasks block network access Findings - Failures compound: a step after a pass passes 70% of the time, after a fail 14% - Agents aren’t running out of time...they’re declaring victory. Only 12% of failed steps got near the time limit, and in 52% the agent reported completion without flagging any failing check Limitations - No isolated-step baseline, so carried-forward errors can’t be separated from step difficulty - Public projects (memorization possible), 38/69 tasks are C/C++, and each model ran in its own harness - Wide CI (11.6–25.7%), so the top models are statistically close My take This benchmark hands agents a precise spec, and they still can’t tell when they’re done. Real work is underdetermined: turning intent into checks is part of the challenge, and almost no benchmark measures it. arxiv.org/abs/2611.07166


























