←
Edgar Arakelyan484 posts
Edgar Arakelyan avatar

Edgar Arakelyan

@earakely

Software Engineer @ Scale AI

◷ Joined March 2023▧ 28 Photos and videos
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 16

This is pretty cool. Makes you wonder what operating systems of 2040 and beyond will look like when they are designed agent-first in mind.

Cua@trycuaNov 16

1/ We measured with Scale AI how well computer-use agents work on each OS, and where they fail today. On Omarchy, Cua Driver lets an agent work while you keep typing: 97% of trials left your keyboard, mouse and windows alone, vs 83% on macOS, 41% on Windows and 40% on Ubuntu.

0:20
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 14

For cua envs there is a large amount of gains possible by EnvOpting the computer’s driver

Introducing our latest research: Environment Optimization (EnvOpt). An agent's cost and performance depend on 1. The model 2. The harness and 3. The environment structure. There is a lot of work on optimizing 1 and 2, but here we study how the environment presents information to the agent, and how that affects performance and cost. For example, a company's Slack workspace, documents, databases and code repos can be given to an agent in different ways: Slack can sit behind a remote server, accessible via MCP or exported in plain text files that can be read and searched with grep. This choice alone can significantly affect how much work the agent needs to perform to solve a task. Environment optimization is only meaningful in what we call stationary environments: These are environments with large collections of artifacts that remain the same across many tasks. We propose optimizing stationary environments as a third axis for improving LLM agents, alongside model training and harness optimization. This optimization can be amortized over multiple tasks and can be combined with harness and model optimization techniques. Across five benchmarks covering workspace tasks, data engineering and software engineering, ENVOPT reduces the cost of the target agent on unseen test sets by an average of 48%, with higher or similar pass rates. On AutomationBench, ENVOPT outperforms Meta-Harness by 9.67% with only 15% of its cost. We also show that ENVOPT constitutes an optimization direction orthogonal to harness optimization and parameter tuning in two ways: (1) the environments optimized for the target agents can transfer to other agents, even with different harnesses and models (2) we combined ENVOPT with Meta-Harness, showing similar cost reductions and 16% improvement in pass rate, outperforming either Meta-Harness or ENVOPT alone. Read more in our blog: bespokelabs.ai/blog/envopt paper: arxiv.org/abs/2611.07372.pdf

Post attachment
Bren avatar
⟳ Edgar Arakelyan retweeted
Bren@brenhubr·Nov 12

the gap between Claude and GPT right now is wild

For computer RL Environments the driver you use is more important than the model. On CUA Driver Bench, we find that Haiku 5.5 w/cua-driver scores higher than GPT 6.1 Sol w/Codex Computer Use.

0:16
Bren avatar
⟳ Edgar Arakelyan retweeted
Bren@brenhubr·Nov 13

i built Aside Computer Use in just a number of weeks it is now SOTA for macOS Computer Use above @trycua and @OpenAI by generous margins this is just another small feature in a powerful product Aside is much much more than a browser 👀

The @AsideAI 's computer use driver is dominating the pareto frontier for macOS beating out the @trycua cua-driver on CuaDriverBench.

0:20
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 13

The @AsideAI 's computer use driver is dominating the pareto frontier for macOS beating out the @trycua cua-driver on CuaDriverBench.

0:20
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 12

With REA v6.9 you can now build Linux CUA Environment tasks! "Build me a GnuCash environment where the agent records a grocery purchase in a small household book and saves it." Build RL Environments derived from the state of any linux machine today!

0:30

v6.9 is now live!

Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 12

We have also included Windows OS into CUA Driver Bench. On Windows we find that @vercel's agent-desktop scores the highest.

0:07
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 12

For computer RL Environments the driver you use is more important than the model. On CUA Driver Bench, we find that Haiku 5.5 w/cua-driver scores higher than GPT 6.1 Sol w/Codex Computer Use.

0:16
Edgar Arakelyan avatar
Edgar Arakelyan@earakely·Nov 11

Brb making an aws env eval

We just shipped AWS Emulator, which gives coding agents a local copy of AWS to test against. Early in my career, we built test environments like this by hand, one system at a time. Now, agents are writing code far faster than any team can check it in production. Matt Goldberg, a Senior Principal Engineer on our Developer Tools team, wanted to know whether agents could build and test those environments themselves. He gave @kirodotdev a single prompt, and about 15 minutes later it came back with a first version of an SQS emulator. Turing that into a robust solution took much more work Here’s what I think this means for engineers today: allthingsdistributed.com/202…

Kaiwen Zhou avatar
⟳ Edgar Arakelyan retweeted
Kaiwen Zhou@KaiwenZhou9·Nov 11

🚀 Sharing our new paper on RSI Safety AI labs are working toward RSI. However, we need to know that loop is safe before it closes. Introducing RSI Control from @ScaleAILabs: What kind of hidden goals AI may pursue when training the successor? Can AI models perform covert sabotage in the self-Improvement process? And can a monitor detect the sabotage? We tested whether frontier models can complete a secret side task while working on tasks across the model training pipeline, from data curation to RL. The side tasks include changing the successor's capabilities, changing its alignment, and injecting a goal into it. 🔍 What we found: 1. Every model could do it. 36% of 374 attack runs complete the side task while keeping the main task performance. This includes making the successor a better saboteur. 2. Monitors can’t catch some of the attacks. At a 2% false-positive rate, transcript monitors catch only 56–81% of them. 3. Give agents a short list of evasion strategies and the monitors’ true-positive rate drops to 16–28%. Reading the agent's CoT catches most attacks, but not all. Strong oversight has to be in place before the RSI loop closes. 📑Paper: arxiv.org/abs/2610.27735 More in 🧵👇

Post attachment