←
Roei Herzig2,618 posts
Roei Herzig avatar✓

Roei Herzig ✓

@roeiherzig

Postdoc @berkeley_ai. Researcher @IBMResearch. PhD @TelAvivUni. Working on Multimodal Robot Foundation Models and Structured Physical AI.

◷ Joined March 2017▧ 192 Photos and videos
Roei Herzig avatar
Roei Herzig@roeiherzig·Jun 18

If you want a vision encoder for dexterous manipulation, what should be the most important part to model? 🤔 Current standard models like CLIP, SigLIP, and DINOv2 have an incredible grasp of semantics and spatial details. But they lack the action-centric structure needed for downstream visuomotor control. But collecting annotated robotic trajectories at scale is SUPER expensive and largely unrealistic. So, how do we bridge this gap? We introduce CAIP (Contrastive Action-Image Pre-training) ⬇️ 🔸 Action-centric upstream: we align visual observations with action chunks through a contrastive objective. 🔸 Human video as a proxy: we represent 3D human hand poses analogously to robotic end-effector actions, tapping into a massive source of human demonstrations. 🔸 Massive scale: pre-trained on over 32,000 hours of manipulation video, driving both sample efficiency and robust generalization. 🔸 Hardware proven: achieves a 76% average success rate on a real-world Dexmate Vega bimanual @DexmateAI manipulator with dual 22-DoF Sharpa Wave hands @SharpaRobotics . 🔸 State-of-the-art: significantly outperforms strong baselines like DINOv2, SigLIP, MVP, and Qwen3.5 ViT across complex tasks, even under unexpected lighting changes and visual distractors. 🌐 Project: caip-encoder.github.io/ 📝 Blog: x.com/roeiherzig/status/2063… 📄 Paper: arxiv.org/abs/2606.17256 💻 Model: huggingface.co/yuvansharma/c…

1:30
Zohar Rimon avatar
⟳ Roei Herzig retweeted
Zohar Rimon@ZoharRimon·Nov 14

Was so fun setting this one up, a great reminder for the wonder of human intelligence. And to anyone who thinks physical intelligence is solved - book a visit to a robotics conference 🤖

Post attachment
0:51
Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 14

“Stuck” in sunny San Diego due to the FAA, so I can finally wrap my head around what an amazing #CoRL2026 it was! Thank you @corl_conf and everyone who made this year’s conference so special. 🧒🤖 We ran our LIVE Child–Robot Challenge for the first time at the conference, and the room was buzzing with curiosity! Huge thanks to @JitendraMalikCV @DJiafei @micoolcho @_amirbar @ZoharRimon @YiruHelenWang James Ni, Frank Hu, @k7agar @hq_fang & @SharpaRobotics for bringing it to life, and to our youngest competitor (my little Adam), the true star of the day. 🎤 And thank you Georgia Chalvatzaki @GeorgiaChal, Jeannette Bohg @leto__jean, Yunzhu Li @YunzhuLiYZ & Mike Rabbat @michaelrabbat for brilliant, inspiring talks. Watch the challenge 👇 spin-workshop.github.io

0:51
Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 13

So, a nationwide ground stop has halted flights... time to keep the robotics party going with another CoRL day at Austin?

Jiafei Duan avatar
⟳ Roei Herzig retweeted
Jiafei Duan@DJiafei·Nov 13

We took part in @JitendraMalikCV’s 5-year-old vs. robot (MolmoAct2) dexterity challenge at the SPIN workshop. The gap is clear: we’re making progress, but robotics is far from solved. Could this be the ultimate test of robot dexterity? 👇

0:27
Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 12

צריך להיות ממש תמים כדי לחשוב שזאת הסיבה העיקרית לעזיבה של אולי המנכל הכי מצליח בעשורים האחרונים. יש פה בטוח דברים אחרים שקורים מאחורי הקלעים.

Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 12

Our child-vs-robot challenge at the #CoRL2026 SPIN workshop starts in 20 minutes (9:40-10:40)! Meet us in Room 209 at 9:35 to walk over together, or head straight to the Griffin Ballroom near the posters.

Post attachment
Georgia Chalvatzaki ✈️ #CoRL2026 avatar
⟳ Roei Herzig retweeted
Georgia Chalvatzaki ✈️ #CoRL2026@GeorgiaChal·Nov 12

Structure can make scale more effective. There is still plenty of science to do! In one hour at #CoRL2026 SPIN workshop, I’ll talk about how robots can make better use of what they know and learn through interaction. And why even MacGyver needs his fingertips 🤖 @leto__jean 😅

Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 12

Tomorrow at @corl_conf: our SPIN workshop! #CoRL2026 🧒🤖 For the first time: a LIVE Child–Robot Challenge with @JitendraMalikCV @DJiafei @micoolcho @ZoharRimon @YiruHelenWang James Ni, Frank Hu, @k7agar @hq_fang, and @SharpaRobotics at 9:40 AM! Plus all-star talks by Georgia Chalvatzaki @GeorgiaChal, Jeannette Bohg @leto__jean, Yunzhu Li @YunzhuLiYZ & Mike Rabbat @michaelrabbat. 📍 8:30 AM, JW Marriott 208 spin-workshop.github.io

Post attachmentPost attachment
Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 11

I started using Carbon very recently, and it doesn’t feel on par with Claude or ChatGPT for standard tasks yet, though it may just need more time.

0:29
Roei Herzig avatar
Roei Herzig@roeiherzig·Nov 11

#CoRL2026 thoughts: I truly believe that for robotic harness to move forward, we need to converge on a much more structured approach, one that doesn’t brute-force the entire search space. We can’t keep spending thousands of dollars to solve a simple task! More on that soon!

0:23