Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
Update: WorldDP is now accepted at Transactions on Machine Learning Research (TMLR) 🚀
Huge thanks to my co-authors, the reviewers, and the editors for the thoughtful feedback that made the paper stronger.
Project page: raktimgg.github.io/worlddp-w…
Preprint: arxiv.org/abs/2606.08775
Code releasing soon.
Introducing WorldDP: A Hierarchical World Model Framework for Multi-Stage Robotic Tasks 🦾✨
tl;dr: high-level world model for subgoal planning + low-level Diffusion Policy for execution + object-centric encoder + particle filter optimization ➡️ multi-stage robotic tasks. 🧵👇
Hierarchical world models, learned end-to-end from pixels!
Excited to introduce H-JEPA: stacked JEPAs, each level in its own embedding space, planning top-down.
Visual AntMaze: 3 levels lift success from 18% to 73% 👇
w/ @kevinghstz, Michael Rabbat, @ylecun, @randall_balestr
Can't make it to #ECCV2026, but our DexWM paper will be there! Swing by Poster #445 in Poster Session 6 of the main conference on Sept 12, where our co-author @JimmyTYYang1 will be presenting it 🚀🤖🦾
Project Webpage: raktimgg.github.io/dexwm/
Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
Introducing LpWM: A Case for Sparse Representations in World Models
Dense Gaussian representations are a choice, not a requirement. We find that sparse representations can make latent dynamics easier to model for planning.
📄arxiv.org/abs/2608.22764
💻github.com/YilunKuang/lpworl…
Scientific terms should have precision. If we use the terms VLM, VLA, WAM in an indiscriminate fashion, as is becoming common in robotics, we are not helping clarity in communication. Let's keep the historical origins of these terms in mind. VLMs arose as multimodal extensions of LLMs-the training was for tasks like VQA (VIsual Question Answering). These capture the static semantics of the scene behind an image. No dynamics. World Models (e.g. @ylecun , Ha & Schmidhuber 2018) on the other hand are primarily dynamics models, which go back to control theory -1960 (Bellman, Kalman etc.) This makes them natural for robotics planning / policies- I am in a state s, what action a should I perform to get to state s'. In classical control, these models were written down a priori by modeling the physics of the system; today we think of them as learned neural networks trained from temporal data e.g. video, robot trajectories. But the concept is the same. We shouldn't mix this concept with VLMs.
Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
Check out our paper and project page for more details:
🌐 Project Page: raktimgg.github.io/worlddp-w…
📚 Pre-Print: arxiv.org/abs/2606.08775
w/ P. Krishnamurthy, @ylecun , Farshad Khorrami
Huge thanks to all my co-authors for their support and guidance.
How WorldDP actually works under the hood:
1️⃣ The Split: Instead of one model doing everything, we split the job hierarchically. The high-level object-centric WM finds feasible subgoals at runtime, and the low-level Diffusion Policy executes them smoothly.
2️⃣ Object-Centric Encoding: We incorporate object-centric representations that decouple environmental entities to aid in learning dynamics and enable sequential planning.
3️⃣ Test-time Planning: We use a Particle Filter (instead of CEM) to handle the multi-modality of robotic actions.
Most visual world models struggle with multi-stage planning. While recent works try to fix this by stacking multiple world models in a hierarchy, we take a different path in WorldDP.
We combine a high-level, object-centric world model to find subgoals with a low-level, goal-conditioned Diffusion Policy to actually reach them.
Introducing WorldDP: A Hierarchical World Model Framework for Multi-Stage Robotic Tasks 🦾✨
tl;dr: high-level world model for subgoal planning + low-level Diffusion Policy for execution + object-centric encoder + particle filter optimization ➡️ multi-stage robotic tasks. 🧵👇
I’m happy to share that I’ve joined @amilabs as a Research Intern for the summer, working on world models! Excited to learn and contribute to the team! 🚀
The lack of a global shutter in ZED cameras is the main reason our group favors RealSense, even though it means we have to develop custom algorithms for features already baked into the ZED SDK. I feel that the ZED SDK, especially its SLAM implementation, is top-tier. Happy to see that the new cameras finally come with global shutter.
🚨 We're giving roboticists a major upgrade to their vision systems!
Today, we announced the launch of the @Stereolabs3D ZED X Nano: a compact, wrist-mount stereo camera engineered for the next generation of robotic manipulation.
“Building on Stereolabs leadership in AI vision and perception solutions, the ZED X Nano allows us to go deeper into the industrial and robotics markets to win new sockets that require smaller form-factor placements,” said Ouster CEO Angus Pacala.
JEPA world models + Hierarchical Planning is a massive step for long-horizon robotics.
A classic failure mode I’ve faced with planning with world models: flat planning often "cheats." For example, in pick-and-place, the robot often reaches the target state in imagination without actually picking the object.
Hierarchical layers fix this by first optimizing for valid subgoals (like the grip) before the finish.
Incredible work, @kevinghstz and team! Huge congratulations. 🚀
The code for DexWM is now publicly available: github.com/facebookresearch/….
The repository includes the full training and evaluation pipelines, along with custom dexterous manipulation datasets generated in RoboCasa, making it easy to reproduce our results and build on top of this work.
Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
Unveiling our new startup Advanced Machine Intelligence (AMI Labs).
We just completed our seed round: $1.03B / 890M€, one the largest seeds ever, probably the largest for a European company.
We're hiring!
[the background image is the Veil Nebula - a picture I took from my backyard, most appropriate for an unveiling]
More details here:
techcrunch.com/2026/03/09/ya…