Okay, so I am making (some) progress with my Computer Use v1 eval suite for @VulcanBench
It's still not there yet, more tasks to add, but already getting some really interesting signal, and experimenting with different ways to share the results.
Here's an early look, note, still need to add harder tasks, review traces in more detail, etc. but as many of you know, I like sharing wayyyyy too early, so here ya go.
Please give me feedback, how do you like this format for a model card? What kind of tasks am I missing from this eval suite that you'd want to know about?
And here it is, our Fable 5.2 benchmark, and a new feature, a thinking report! 🧠
Can confirm, no better way to start a Saturday than a fresh benchmark 🖖
Okay, the day has come, my Fable 5.2 Benchmark with @VulcanBench is now complete 🖖
This is the first model to clear every task, and thus, am calling it, "the Frontier v4 killer"
And honestly, if something was going to kill this eval suite, it's an honor to have Fable be the one to do it.
That being said, VulcanBench is all about testing models on real engineering tasks, that teams like mine would use a model for every day, to really understand first, if you need the model at all, and second, what effort level you need if you do.
I say anything in the 80 - 90 range on Frontier v4 is strong enough for daily work, and that is why, if you use Fable 5.2, Low effort is likely all you need for routine daily engineering tasks.
But you'll pay a price, as you can see from the cost-per-task chart, it isn't cheap. But it does think in amazing ways.
Which is why I'm also introducing a brand new section on VulcanBench where I share analysis on how a model thinks...and yeah, hard to think of a better model to kick this off with than Fable.
With that, here's how you access the benchmark and the thinking report if you want to do a deeper dive into either or both.
VulcanBench on Frontier v4:
vulcanbench.com/benchmarks/s…
How Fable 5.2 Thinks Report:
vulcanbench.com/reports/01-h…
Live long and benchmark 🖖
Might delay this by a day or two, still reviewing traces, and putting together a new deliverable from VulcanBench that I'm probably going to call:
Thinking Reports 🧠 📄
Looking at the traces for a model like Fable is something that for me, has become in many ways like appreciating art.
That might sound overly dramatic, but it really does feel that way.
And while I know, of course the most interesting thing to most ppl is the actual benchmark report, i.e. showing how a model performs from an accuracy, cost, and timing standpoint...maybe there are other people like me that want to understand more about how the model thinks when it's solving a task in an eval suite, and how this varies from model-to-model.
I can think of no better model to start this "Thinking Reports" project with, than Fable 5.2.
So expect to see this sometime over the weekend, going to take more time to really do this right.
Live long and study how models think 🖖
Fable 5.2 benchmark results are in, reviewing the traces in detail now.
Publishing model cards in the morning.
You thought Ocarina of Time was a masterpiece, wait until you see the traces for this one 🖖
Fable 5.2 benchmark results are in, reviewing the traces in detail now.
Publishing model cards in the morning.
You thought Ocarina of Time was a masterpiece, wait until you see the traces for this one 🖖
Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.
In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.
All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).
Early results with Fable 5.2, and wow, just wow, this is one heck of a model.
After seeing how it does at Low effort, I think it's safe to say, this is the model that will retire our Frontier v4 eval suite, it's been a good ride, now to finish Frontier v5 🖖
Just got access to Fable 5.2 today, and immediately started benchmarking on @VulcanBench w/Frontier v4, safe to say, I think Fable 5.2 will mark the official sunsetting of this eval suite.
I've been working on Frontier v5 for over a month, getting closer, but not there yet.
That being said, just look at Fable 5.2 on Low effort compared to Fable 5.1. From ~26 min/task down to ~5 min/task, and from $183 to $41, this is a truly amazing model. Can't wait to do a deeper dive into the traces to see how it thinks under-the-hood.
More to come, this is just the first effort level, but I think Fable 5.2 will very likely get a perfect score on every one, which means, it's the model that puts Frontier v4 into retirement.
It was a good eval suite, but it has met it's match.
Live long and benchmark 🖖
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
Okay, no better way to start a Friday than with some fresh benchmark results right?
When I first ran my Muse Wonder Full benchmark with @VulcanBench I didn't have enough credits to run Max.
But my subscription renewed, and I was able to finish the benchmark and run Max, so now have the complete benchmark finished, 115 runs, 23 tasks each.
Two nuggets 🔽
1. Increasing effort level doesn't really make much of a difference in accuract: Muse Wonder Full scores between 87.0 and 88.4 at every level. Going from Low to Max adds about 1.4 points but takes 50% longer (27 to 41 minutes) and costs 50% more (about $3.93 to $5.91 a task).
2. GPT-6.1 Sol pretty much ties it for a lot less. At Max, Sol scores 88.3 vs Muse Wonder Full’s 88.4, at about a thirteenth of the cost ($0.46 vs about $5.91 a task) and in about a third of the time (15 vs 41 minutes).
My final assessment now that the data is in: the Meta DS team has made awesome progress with Muse, massive improvements, but still not a model I would use for routine daily coding tasks when I can get the same accuracy, faster and cheaper with a model like GPT-6.1 Sol.
All data live on the site now:
vulcanbench.com/benchmarks/s…
And as always, all the traces, available on Github:
github.com/morganlinton/vulc…
Live long and benchmark 🖖
None of this is built yet, still in the planning phases, and safe to say I have a lot to learn as I go here.
If you run computer-use agents, I would love to hear more about what bugs you the most and where you think I might need to go deeper with this eval.
And if there's an existing benchmark you think does a really good job already with Computer Use, please let me know, I'd love to learn from them too!
Live long and make more benchmarks 🖖
Adding two new numbers that I track:
1. screenshot tax (pass rate with pixels only vs. pixels plus an accessibility tree)
2. a safety track (injection compliance and unsafe-action rates, reported next to task success).
I'm starting with three key design rules for this eval suite:
1. Grade the end state, not the path to the end state.
2. Build difficulty into the world, i.e. add some fun distractions, multi-screen dependencies, interrupting dialogs, etc.
3. Before a task gets in, I try to cheat it and if the checker still says pass, I'll know the checker is wrong. I think this will be an important rule for correctly assessing computer use behavior.
First, the design doc is up for VulcanBench ComputerUse v1, I'm all for getting feedback on how to make it better.
My goal is: measure whether a model can reliably and safely operate software it has never seen, through the products people actually use.
I have been looking at expanding VulcanBench beyond agentic coding, and now have a clear direction I want to move in: Computer Use.
Personally, I think Computer Use is going to be one of the most useful things we do with agents over the next 3-5 years, and I also think it's something we'll want to do as fast and token-efficiently as possible, without sacrificing accuracy.
Here's a bit more about what I'm thinking, a "Morgan decided to build an eval suite for Computer Use" 🧵
Woke up thinking about model routing, and the direction many companies are going.
Today most individuals and small(ish) teams use subscriptions with pretty heavily subsidized token costs.
Larger companies moved to Enterprise pricing which is often 5x - 10x what individuals or small teams pay.
This has led to the creation of model routers, essentially solutions that optimize the models engineering teams use and save 30% - 50% off Enterprise pricing.
Often, the key unlock here is for large teams, a lot of engineers just use one model for planning, coding, and code review, often at High effort, because they don't want to think about things like model and effort level.
Model routers often use different models, and effort levels at each of these stages, and (maybe) change what they use based on task size/difficulty/language, but I'm not so sure about that.
Starting to noodle on evaluating model routers to learn two key things:
1. Are model routers actually changing model and effort level based on the complexity of the task, size of the repo, programming language, etc. or do they just use the same models/effort levels for everything?
2. Sure, you're saving off of Enterprise pricing if you have hundreds of engineers just using the same model at High effort (or higher) for everything, but how much more are you paying vs using a subscription?
Had a good convo with Fable about this, put together a plan, starting to build. More to come 🖖