1/ We measured with Scale AI how well computer-use agents work on each OS, and where they fail today.
On Omarchy, Cua Driver lets an agent work while you keep typing: 97% of trials left your keyboard, mouse and windows alone, vs 83% on macOS, 41% on Windows and 40% on Ubuntu.
Introducing our latest research: Environment Optimization (EnvOpt).
An agent's cost and performance depend on
1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost.
For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task.
Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques.
Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms
Meta-Harness by 9.67% with only 15% of its cost.
We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways:
(1) the environments optimized for the target agents can transfer to other agents, even with different
harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone.
Read more in our blog: bespokelabs.ai/blog/envopt
paper: arxiv.org/abs/2611.07372.pdf
For computer RL Environments the driver you use is more important than the model.
On CUA Driver Bench, we find that Haiku 5.5 w/cua-driver scores higher than GPT 6.1 Sol w/Codex Computer Use.
i built Aside Computer Use in just a number of weeks
it is now SOTA for macOS Computer Use above @trycua and @OpenAI by generous margins
this is just another small feature in a powerful product
Aside is much much more than a browser 👀
With REA v6.9 you can now build Linux CUA Environment tasks!
"Build me a GnuCash environment where the agent records a grocery purchase in a small household book and saves it."
Build RL Environments derived from the state of any linux machine today!
For computer RL Environments the driver you use is more important than the model.
On CUA Driver Bench, we find that Haiku 5.5 w/cua-driver scores higher than GPT 6.1 Sol w/Codex Computer Use.
We just shipped AWS Emulator, which gives coding agents a local copy of AWS to test against.
Early in my career, we built test environments like this by hand, one system at a time. Now, agents are writing code far faster than any team can check it in production. Matt Goldberg, a Senior Principal Engineer on our Developer Tools team, wanted to know whether agents could build and test those environments themselves. He gave @kirodotdev a single prompt, and about 15 minutes later it came back with a first version of an SQS emulator. Turing that into a robust solution took much more work
Here’s what I think this means for engineers today: allthingsdistributed.com/202…
🚀 Sharing our new paper on RSI Safety
AI labs are working toward RSI. However, we need to know that loop is safe before it closes.
Introducing RSI Control from @ScaleAILabs: What kind of hidden goals AI may pursue when training the successor? Can AI models perform covert sabotage in the self-Improvement process? And can a monitor detect the sabotage?
We tested whether frontier models can complete a secret side task while working on tasks across the model training pipeline, from data curation to RL. The side tasks include changing the successor's capabilities, changing its alignment, and injecting a goal into it.
🔍 What we found:
1. Every model could do it. 36% of 374 attack runs complete the side task while keeping the main task performance. This includes making the successor a better saboteur.
2. Monitors can’t catch some of the attacks. At a 2% false-positive rate, transcript monitors catch only 56–81% of them.
3. Give agents a short list of evasion strategies and the monitors’ true-positive rate drops to 16–28%.
Reading the agent's CoT catches most attacks, but not all.
Strong oversight has to be in place before the RSI loop closes.
📑Paper: arxiv.org/abs/2610.27735
More in 🧵👇