←
Eric Chan85 posts
Eric Chan avatar

Eric Chan

@ericryanchan

Chief Scientist of Rhoda AI prev. PhD Student, Stanford University

◷ Joined December 2020▧ 7 Photos and videos
Eric Chan avatar
Eric Chan@ericryanchan·Mar 10

Today, we announce our team’s progress in pursuing a different type of foundation model for robotics: the Direct Video Action Model (DVA), which does our best to take robotics and turn it into a generative modeling problem we can scale. Technical blog: rhoda.ai/research/direct-vid…

Shitao Tang avatar
⟳ Eric Chan retweeted
Shitao Tang@TsT87648373·Nov 14

Many friends at CoRL worry that frontier models like Astra and Claude will take over robot control. I want to offer a different perspective. I see three routes: VLA: Start with vision-language pretraining, then train on actions. Transfer semantic and spatial understanding into robot capability. WAM: Start with video-generation pretraining, then learn actions. Transfer knowledge of how the world changes into robot capability. Astra/Claude: Bring broader visual, language, and coding capabilities to robot control. Their 3D understanding—and how much of it transfers to manipulation—is striking. My hypothesis is that learning across code and visual domains, such as Blender tasks, contributes to this capability. However, three questions remain for Astra/Claude: 1. Can latency fall enough to support responsive, closed-loop control? 2. How much capability remains with less chain-of-thought reasoning? 3. Can we distill that capability into a smaller, faster model without losing its generality? These are all questions about the cost of turning impressive capability into a practical robot policy. I expect progress on all three, though how much remains uncertain. The same optimism applies to WAMs and VLAs: I expect them to improve with scale. History tells us that the model that turns data and parameters into capability most efficiently will win. Astra and Claude show what’s possible at scale. Which approach gets there most efficiently is still an open question. The competition is just beginning. At Rhoda, we’re betting on WAMs, and our results give us concrete reasons to believe in their data and parameter efficiency [1] [2] [3]. We’re hiring across multiple roles. Join us to build the next generation of robot intelligence.rhoda.ai/career… [1] rhoda.ai/research/direct-vid… [2] x.com/rhodaai/status/2099914… [3] x.com/rhodaai/status/2117663…

At Rhoda, we care deeply about the science of pre-training for robotics. In one of the most rigorous studies of its kind, over thousands of trials and hundreds of hours of robot evaluations, we show how scaling web-video pre-training leads to better real-world robot performance.

0:21
Joanne Truong avatar
⟳ Eric Chan retweeted
Joanne Truong@joannetruong·Nov 3

(1/n) It’s been really exciting to bring mobility into our Direct Video-Action (DVA) model! Here’s how a video model learns to drive, steer around obstacles, and adapt when the destination moves, from just 15 minutes of driving data. One policy, no maps, no navigation stack 🧵

We taught a robot how to move around in only 15 minutes. That’s all the mobility data our Direct Video-Action-powered robot needed to accomplish a task that requires mobility and manipulation.

2:22
Eric Chan avatar
Eric Chan@ericryanchan·Nov 3

The best systems are the ones that are as simple as possible. In this case, we perform mobile manipulation with a single DVA. No separate mobility or navigation stack; just pure joint prediction of both how the arms move and how the robot maneuvers around novel obstacles. Impressive work, @joannetruong!

We taught a robot how to move around in only 15 minutes. That’s all the mobility data our Direct Video-Action-powered robot needed to accomplish a task that requires mobility and manipulation.

2:22
Vincent Sitzmann avatar
⟳ Eric Chan retweeted
Vincent Sitzmann@vincesitzmann·Oct 20

Thrilled to introduce Mirror, our first "learning-from-experience" project! Assume you had a video generative model that, no matter your situation, could always generate video of what it would look like to solve a task. How can you put that plan into action? Clearly, you need to learn some video-to-action mapping - but where do you get the data for that? It turns out you can collect that data in an amazingly sample-efficient way by *incrementally following the video planner*! (1/n)

If video models scale like LLMs, they'll soon generate a perfect video plan for any embodied task. Do we need anything else? Introducing Mirror, which learns to follow these plans just by practicing, and even crafts an iron pickaxe in Minecraft w/o rewards or action labels! (1/n)

1:15
Eric Chan avatar
Eric Chan@ericryanchan·Oct 19

The result is a new way of learning representations that is stable, informed by the theory, and most importantly, scalable. And it's a general recipe, so maybe we'll start to see some of these techniques applied to new domains and modalities!

Eric Chan avatar
Eric Chan@ericryanchan·Oct 19

One key insight is a connection to contrastive learning. Contrastive methods learn to distinguish pairs of observations that occur together from randomly paired observations. The optimal score for this task is pointwise mutual information (PMI): a measure of how much more likely two observations are to occur together than by chance. Rather than learning PMI indirectly through contrastive learning, our method uses likelihoods from a generative model to estimate PMI and directly supervise the representation!

Eric Chan avatar
Eric Chan@ericryanchan·Oct 19

Excited to share @BaadeAlan's latest work: a new approach to representation learning designed specifically for generative modeling! Semantic representations like DINO have become remarkably effective tokenizers for image and video generation. But DINO was never designed for this task. Its success in generative modeling is almost accidental. What if we learned representations specifically to make generation easier?

1/ The tokenizer is a new scaling axis! We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse. We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI Thread, Paper, and Blogpost below

0:29
Vincent Sitzmann avatar
⟳ Eric Chan retweeted
Vincent Sitzmann@vincesitzmann·Oct 19

Very excited about this one: a completely new representation learning objective, with a proper theory of what it converges to: the representation of an image converges to features such that their dot product with another image's representation is their pointwise mutual information! @BaadeAlan knocked it out of the park :) (1/n)

1/ The tokenizer is a new scaling axis! We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse. We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI Thread, Paper, and Blogpost below

0:29
Leo Dong avatar
⟳ Eric Chan retweeted
Leo Dong@leooeld·Oct 14

(1/n) Watch our robot search a box and find textured balls, entirely without vision! Over the past few months, it’s been incredibly fun to work on bringing touch into our Direct Video-Action (DVA) model. Here’s how our video model is “seeing” through touch 🧵

What can a robot do without vision? We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded. Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

2:53
Vincent Sitzmann avatar
⟳ Eric Chan retweeted
Vincent Sitzmann@vincesitzmann·Oct 14

Touch is essential to human manipulation. Yet, we have so far not had a tactile sensor that can do what human skin can: sense texture, firmness, vibration at a viable form factor. At Rhoda, we have built a new sensor in-house that can do all of these things, while being much much smaller than commercially available ones!

What can a robot do without vision? We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded. Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

2:53
Eric Chan avatar
Eric Chan@ericryanchan·Oct 14

I often get asked how world models will integrate tactile data, when they’re trained primary from video. Here’s a simple solution that works surprisingly well: embed the tactile information into the video stream, as pixels. At Rhoda, we’ve developed inexpensive and sensitive tactile sensors, and have demonstrated their capabilities, including allowing a blind robot to sense. It’s an exciting move towards greater dexterity!

What can a robot do without vision? We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded. Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

2:53
Rhoda AI avatar
⟳ Eric Chan retweeted
Rhoda AI@RhodaAI·Oct 6

Continuing our RhodaAtWork series with another look at Direct Video-Action (DVA) in action. This time, a single model handles multiple tasks across two home appliances: refrigerator EPS foam installation and washing machine drain hose routing.

0:50
Rhoda AI avatar
⟳ Eric Chan retweeted
Rhoda AI@RhodaAI·Oct 1

More industries. More real customer work. Introducing RhodaAtWork, a series on the customer workflows our robots are learning to handle across manufacturing and logistics. First up: unpacking, scanning and stacking component reels for electronics manufacturing.

2:16
Stephen James avatar
⟳ Eric Chan retweeted
Stephen James@stepjamUK·Sep 15

Rhoda AI checked whether scaling web video pretraining actually helps real robots, using a real customer task. Unpacking 10kg boxes of bearings and sorting the packaging. Turns out yes. Bigger models do better, more pretraining compute does better, and the compute gap is widest when you don't have many robot demos. Refreshing to see a company display this amount of rigour. Rare these days.

At Rhoda, we care deeply about the science of pre-training for robotics. In one of the most rigorous studies of its kind, over thousands of trials and hundreds of hours of robot evaluations, we show how scaling web-video pre-training leads to better real-world robot performance.

0:21
Eric Chan avatar
Eric Chan@ericryanchan·Sep 15

We recently came out with our blog post, Does Scaling Web-Video Pre-training Help Real Robots Do Real Work? This is a pretty exciting moment for robotics, because as far as I am aware, it is the first evidence that robot foundation models can scale on internet video data; not merely on teleoperation, UMI data, or egocentric data, but on the data you can truly find anywhere. It’s the necessary requirement for reaching massively powerful models.

At Rhoda, we care deeply about the science of pre-training for robotics. In one of the most rigorous studies of its kind, over thousands of trials and hundreds of hours of robot evaluations, we show how scaling web-video pre-training leads to better real-world robot performance.

0:21