←
Rui Wang137 posts
Rui Wang avatar

Rui Wang

@theruiwang

Research Lead @yutori_ai. Prev, post-training lead of Llama 3 multimodal @AIatMeta. Earlier, @FAIR, @CarnegieMellon

◷ Joined September 2016▧ 12 Photos and videos
Rui Wang avatar
Rui Wang@theruiwang·Aug 26

Excited to share that Navigator n2 is officially out! The core idea behind n2 is simple: models should use computers the computer way, not the human way. There’s no single best interface. GUI, CLI, tools, code — pick the right one at every step. The hard part is making those choices well over long horizons: knowing when to switch, when not to, and how to keep making progress over hundreds of steps. The training loop ended up following the same idea: training computer-use models is itself a computer-use task. We use computer-use agents to explore applications, create tasks, test verifiers, find reward hacks, edge cases, and where the latest model fails. Those failures shape the next round of training data. It creates a pretty natural recursive loop: better models make better data, which makes better models. The result is a frontier-level model at a fraction of the cost— 65.2% on OSWorld 2.0 at $1.46 per task. Very proud of the team and where n2 landed!

Post attachment

Today we’re introducing Navigator n2. It’s a frontier computer-use model, with just 27B parameters.

2:03
Rui Wang avatar
Rui Wang@theruiwang·Oct 11

(life) probably not many sim racers have tried this: Codex Computer Use helped me configure AC on a fresh Windows install, with me handling the physical part. we got this list done in about an hour: ✓ content manager, CSP & Pure LCS installation ✓ weather controller and PP filter configs ✓ wheel, pedal and shifter mapping ✓ steering and pedal calibration ✓ display settings ✓ driving assist configs ✓ force feedback tuning ✓ perf benchmarks and config backups turned a notoriously tedious setup into an efficient & enjoyable process. highly recommended

Post attachment
Jason Wei avatar
⟳ Rui Wang retweeted
Jason Wei@_jasonwei·Oct 6

Visiting SF for COLM and curious what SF is all about? The bad news: COLM is in the dodgiest neighborhood and if you hang around there, you’ll hate SF just like I used to. The good news: your twitter buddy Jason Wei wrote a guide on experiencing SF like a local: jasonwei.net/blog/the-best-t…

Post attachment
Carnegie Mellon University avatar
⟳ Rui Wang retweeted
Carnegie Mellon University@CarnegieMellon·Sep 30

Today, Carnegie Mellon University and Ken Griffin, founder and CEO of Citadel and founder of Griffin Catalyst, announced a historic $3 billion gift to elevate Carnegie Mellon in Pittsburgh and launch Carnegie Mellon University Miami. #CMUMiami Learn more: cmu.edu/news/stories/archive…

1:42
Jing Yu Koh avatar
⟳ Rui Wang retweeted
Jing Yu Koh@kohjingyu·Oct 1

Computer-use agents are extremely good! But they need to improve on a metric which most benchmarks don't track. How much time do CUAs take to complete a task? Introducing cua-speedrun: a benchmark for accuracy, speed, and cost under matched infra. cuaspeedrun.com 👇🧵

1:07
Rui Wang avatar
Rui Wang@theruiwang·Sep 18

8/ To sum up: Two years ago, “computer use” named a research problem: can a model use a computer like us? Now it can. That was a milestone, not the destination. A computer offers much more than its screen. Using a computer should mean using ALL of it. More in: x.com/theruiwang/status/2099…

Rui Wang @theruiwang Sep 14 Article The Other Hand tldr: Computer use = coding + clicking In 2024–25, people debated whether computer use would work. In 2026, the question is: what's next for computer use? But before answering that, we need to ask

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

7/ Computer use will advance on two frontiers: The first frontier is open-ended knowledge work. The bottleneck is quality, not compute: valuable tasks require domain expertise from task design through verification. Coding environments already sell for thousands of dollars; realistic end-to-end knowledge work environments could soon be worth hundreds of thousands. The second is digital automation at scale. This one is getting less attention than it should. Much valuable digital work is made of small, repetitive tasks too variable for scripting, like validating promo codes across O(10k) websites. Each task may be worth only cents, but at scale the value is enormous. The challenge is frontier-level performance at a cost that justifies the business. Typesafe AI is a great attempt at the latter. Excitement aside, I’m sure some folks have already realized the limitations when it comes to computer use—the problems aren’t new. I might expand on this separately.

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

6/ How do we train computer-use agents going forward? So we merge coding and clicking data, are we done? No. Today's models are already pretty good at coding and clicking. The harder problem is teaching them when to use what. They can code. They can click. They can’t choose. And choosing has to be trained where it matters. If a task can be finished through one interface, the model learns nothing about when to reach for the other, no matter how many such tasks it sees. What we need are realistic environments that naturally require both. Luckily, much of knowledge work is exactly this flavor.

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

5/ If that is what computer use means, GUI agents are over. They do everything the slow way. But moving away from GUI agents does not mean moving away from the GUI. Clicking still matters. Much of the software we use still has no API, and some of it may never have one. Until that changes, the GUI remains the generalization guarantee.

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

4/ Computer use should mean using the WHOLE computer. Graphical and programmatic interfaces are both just ways to interact with a computer. The best way to use a computer is to choose the best interface at each step, and switch between them seamlessly. In 2026, this is what I think computer use should mean: Computer use = coding + clicking Same for computer-use agents (CUA): CUA = agents that can code and click And agents that only click are GUI agents.

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

3/ GUIs generalize incredibly well, but clicking only isn't how we solve real-world problems. If a human can use some software via GUIs, an agent can at least use it the same way. But generalization doesn't mean efficiency—and sometimes not accuracy either. Think about doing our taxes, a task we all want to automate. An agent doing it well might pull transactions from banks through APIs, extract numbers from PDFs, write a script to categorize expenses and check the math, then enter everything into tax software through the GUI. Remove any one of them, and the task becomes either much slower or even impossible.

Rui Wang avatar
Rui Wang@theruiwang·Sep 18

2/ Computer use research began by teaching models to use computers like we do: look at a screen, move a mouse, type on a keyboard. It's a natural place to start. After all, that’s how we use computers. And the progress is fast! GUI grounding is getting very strong, and tasks that take a person several to tens of minutes—sometimes even a few hours—are effectively solved now.

Post attachment
Rui Wang avatar
Rui Wang@theruiwang·Sep 18

1/ Computer-use agents ≠ GUI agents. Clicking is only one way to use a computer. The best agents will code, call tools, use the CLI, and click—using everything a computer provides, and choosing the right interface for each step.

Pengchuan Zhang avatar
⟳ Rui Wang retweeted
Pengchuan Zhang@PengchuanZ·Sep 8

As a PhD in mathematics who now works in AI/ML, the recent Navier–Stokes news may bring both excitement and unease to the math community. My PhD advisor, Yizhao T. Hou, has spent more than two decades studying blowup in the Euler equations with his students and collaborators.