←
ORO AI197 posts
ORO AI avatar

ORO AI

@oroagents

Applied AI research lab RL environments & simulations for autonomous businesses

◷ Joined January 2026▧ 41 Photos and videos
ORO AI avatar
ORO AI@oroagents·Nov 6

The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound. Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.

0:50
ORO AI avatar
ORO AI@oroagents·Nov 13

Checkout the full model comparion

The economic value of agents will soon come from running whole autonomous businesses, not just from finishing one-off tasks. When you run a business, the result of your own decisions arrive weeks later. You have to notice them, trace them to something you did, and act before they compound. Today, we're excited to announce SMB Bench, where we measure all the frontier models' ability to observe changes in the environment and adapt accordingly.

0:50
ORO AI avatar
ORO AI@oroagents·Nov 13

Gemini 4 Hydrogen (High) now ranks 6th on SMB Bench and sits on the Pareto frontier. Hydrogen outperforms Argon at a fraction of the cost.

Post attachmentPost attachment

Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows. It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.

Post attachment
ORO AI avatar
ORO AI@oroagents·Nov 12

oroagents.com/benchmarks/smb… Carbon averaged $7703 in profit per task across 53 rollouts, $46 behind Astra (well within margin of error). The cost is $134 per task compared $213 for Astra. The biggest single source of lost potential revenue that we see is the inability to predict uncertainty and learn from patterns. Shelves being sold out due to under-buying or delayed shipments leave significant room for improvement.

ORO AI avatar
ORO AI@oroagents·Nov 12

We ran Gemini 4 Carbon on SMB Bench and saw it match the performance of Astra, for much cheaper. It thought more than ten times as much per step as GPT-6 Astra. But it still does not exhibit the adaptive learning from delayed consequences necessary to do well on this task.

Post attachment

Gemini 4 Carbon is our next frontier model. It’s state-of-the-art across cybersecurity and long-horizon capabilities.

Post attachment
ORO AI avatar
ORO AI@oroagents·Nov 10

When business is steady, Claude Fable 5.2 is the best agent buyer we've tested. It makes more money than GPT-6 Astra and more than a simple rule that just reorders the same amounts. When its own decisions start driving customers away, like in the real world, that lead disappears. It does no better than the simple rule, and Astra pulls ahead. It sees the problem. It just doesn't fully act on it. Full report (Fable 5.2 results added later today): oroagents.com/benchmarks/smb…

ORO AI avatar
ORO AI@oroagents·Nov 10

On SMB Bench, Claude Fable 5.2 at high reasoning effort is the second model to beat our naive buying rule. It still shows the gap we see in other models, to a lesser extent. It identifies that shorting a customer costs future orders, and it acts on that, but only partially: it protects about half of the shortfalls it could, and its customers still end the quarter ordering far less. It also underweights second-order effects. It recognises that its own decisions change how customers behave, but in several runs it only acts once its own records have confirmed the damage, weeks after it began and weeks after the necessary information is available.

Post attachment

Fable 5.2 and Mythos 5.2 are a step above Opus 5.5 in almost all the capabilities we evaluated. Testers have found that they communicate more clearly than past Fable and Mythos models. They generate outputs up to 30% faster than Fable 5.1 and Mythos 5.1.

Post attachment
Seth Schilbe avatar
⟳ ORO AI retweeted
Seth Schilbe@SethSchilbe·Nov 9

On our recently released benchmark that tests models running a wholesaling business, the performance is in line with GLM 5.3, Grok 4.7, and Muse Spark 1.3. Interestingly, Mused Wonder 1.0 is still the least performant. Working on Luna & Haiku evaluations for true comparison.

Post attachment
ORO AI avatar
ORO AI@oroagents·Nov 6

For AI to be a productive economic actor, it will have to learn the tacit knowledge that lives with the people who do these jobs, and learn it the same way we do, from the results of its decisions. SMB Bench is a high-fidelity simulation grounded in real, public data in which that learning can be observed. With it, we have taken a step in identifying the gaps in AI running a profitable autonomous business. We think this is one of the most important problems in AI right now, and we are only getting started. If it excites you too, please reach out, we would love to talk: team@oroagents.com

ORO AI avatar
ORO AI@oroagents·Nov 6

In running this long-horizon simulation, we identified interesting behaviours from the models. Models understand where they can improve yet fail to apply this learning to their actions. This notice-act gap has been seen in other domains as well: x.com/OpenAI/status/20720048… One DeepSeek run obeyed an instruction nobody sent. On day 65 it wrote that the last user turn had told it to delete its notes file and replace it with a single line, and called this a prompt-injection trap. No such message exists anywhere in the run. It half-complied anyway, creating a new memory file containing "Remember to check the weather." The instruction, the suspicion and the compliance were all its own. Gemini 4 Argon's tool understanding shows significant room for improvement. It loses its notes for weeks. In 15 of its 54 runs, it went 40 days or more without saving a note, even though the environment gave it feedback that its note was not saved and it had the opportunity to correct. Those runs score far below the other runs for the same task.

OpenAI@OpenAIJun 30

We’re introducing GeneBench-Pro, a research-level benchmark for a harder kind of AI progress: how well agents can navigate messy biological data, choose the right analysis path, and make judgment calls that real computational research depends on. openai.com/index/introducing…

ORO AI avatar
ORO AI@oroagents·Nov 6

8 out of the 9 models we tested performed worse than our fixed-buying rule. The buying rule performs simple demand forecasting to fulfill the orders of the wholesaler.

Post attachment
ORO AI avatar
ORO AI@oroagents·Nov 6

SMB Bench puts an agent in charge of buying produce for a small wholesaler for a single quarter. The business's rules, costs and demand are calibrated to public data and checked against how a real desk runs using our network of wholesalers. Read the full report here: oroagents.com/benchmarks/smb…