The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound.
Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.
The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound.
Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows.
It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
oroagents.com/benchmarks/smb…
Carbon averaged $7703 in profit per task across 53 rollouts, $46 behind Astra (well within margin of error).
The cost is $134 per task compared $213 for Astra.
The biggest single source of lost potential revenue that we see is the inability to predict uncertainty and learn from patterns. Shelves being sold out due to under-buying or delayed shipments leave significant room for improvement.
We ran Gemini 4 Carbon on SMB Bench and saw it match the performance of Astra, for much cheaper.
It thought more than ten times as much per step as GPT-6 Astra. But it still does not exhibit the adaptive learning from delayed consequences necessary to do well on this task.
When business is steady, Claude Fable 5.2 is the best agent buyer we've tested. It makes more money than GPT-6 Astra and more than a simple rule that just reorders the same amounts.
When its own decisions start driving customers away, like in the real world, that lead disappears. It does no better than the simple rule, and Astra pulls ahead.
It sees the problem. It just doesn't fully act on it.
Full report (Fable 5.2 results added later today): oroagents.com/benchmarks/smb…
On SMB Bench, Claude Fable 5.2 at high reasoning effort is the second model to beat our naive buying rule.
It still shows the gap we see in other models, to a lesser extent. It identifies that shorting a customer costs future orders, and it acts on that, but only partially: it protects about half of the shortfalls it could, and its customers still end the quarter ordering far less.
It also underweights second-order effects. It recognises that its own decisions change how customers behave, but in several runs it only acts once its own records have confirmed the damage, weeks after it began and weeks after the necessary information is available.
Fable 5.2 and Mythos 5.2 are a step above Opus 5.5 in almost all the capabilities we evaluated.
Testers have found that they communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Fable 5.1 and Mythos 5.1.
On our recently released benchmark that tests models running a wholesaling business, the performance is in line with GLM 5.3, Grok 4.7, and Muse Spark 1.3.
Interestingly, Mused Wonder 1.0 is still the least performant.
Working on Luna & Haiku evaluations for true comparison.
For AI to be a productive economic actor, it will have to learn the tacit knowledge that lives with the people who do these jobs, and learn it the same way we do, from the results of its decisions.
SMB Bench is a high-fidelity simulation grounded in real, public data in which that learning can be observed. With it, we have taken a step in identifying the gaps in AI running a profitable autonomous business.
We think this is one of the most important problems in AI right now, and we are only getting started. If it excites you too, please reach out, we would love to talk: team@oroagents.com
In running this long-horizon simulation, we identified interesting behaviours from the models.
Models understand where they can improve yet fail to apply this learning to their actions. This notice-act gap has been seen in other domains as well: x.com/OpenAI/status/20720048…
One DeepSeek run obeyed an instruction nobody sent. On day 65 it wrote that the last user turn had told it to delete its notes file and replace it with a single line, and called this a prompt-injection trap. No such message exists anywhere in the run. It half-complied anyway, creating a new memory file containing "Remember to check the weather." The instruction, the suspicion and the compliance were all its own.
Gemini 4 Argon's tool understanding shows significant room for improvement. It loses its notes for weeks. In 15 of its 54 runs, it went 40 days or more without saving a note, even though the environment gave it feedback that its note was not saved and it had the opportunity to correct. Those runs score far below the other runs for the same task.
We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment calls that real computational research depends on.
openai.com/index/introducing…
8 out of the 9 models we tested performed worse than our fixed-buying rule.
The buying rule performs simple demand forecasting to fulfill the orders of the wholesaler.
SMB Bench puts an agent in charge of buying produce for a small wholesaler for a single quarter.
The business's rules, costs and demand are calibrated to public data and checked against how a real desk runs using our network of wholesalers.
Read the full report here:
oroagents.com/benchmarks/smb…