How? Contrastive learning on raw video, no labels:
➕ Positive: optical flow finds what moves; we cut it out and paste it into another scene. Same action, new scene → pull together.
➖ Hard negative: another moment of the same video. Same scene, new action → push apart.
One still image, many actions: start from one anime-style robot picture and replay the arm motions of three robot demos (mug onto a plate, book onto a shelf, sweeping), just by swapping the clip. 🎨
And every action works in every world: 6 camera motions × 4 imagined worlds.
Robots, too. Demonstrations from robot arms (UT Austin MUTEX, Berkeley UR5) drive the arm in the LIBERO simulator, and motions carry over between LIBERO and RLBench. 🤖
Large video: generated. Inset: the source clip.
Give OnlyAct a source clip: a racing game, a robot demo, a shooter. It reads the action from each pair of frames, and a world model plays it out in a new scene, here from a single image.
Forza → kart track 🏁
LIBERO → unseen MetaWorld arm 🤖
Valorant → lantern-lit alley
🎥Reshoot a video from a camera that was never there~
Excited to share #Holo4D, accepted to #neurips2026 !
Give it a video and a new camera path: Holo4D lifts the video into 3D tracks and uses them to steer a video model. The camera moves; what happened stays the same. 🧵
We release the data: 38,780 two-player Minecraft episodes with 8 synchronised views and 19,775 two-player Doom episodes with 10 views.
🤗 huggingface.co/collections/l…
It works in Minecraft, in Doom and on real multi-camera capture: places stay where they were when a player returns, and what one player does is seen by the other.
Fixed cameras barely change, so our δ-VAE keeps each camera's first frame and codes only what changes: 8× fewer tokens per camera. The agents' views stay full frames.
OverSeer then generates all views together, frame by frame, from the agents' actions and camera poses.
A creeper explodes while both agents look away.
A world model built on egocentric views alone loses the crater. OverSeer's fixed cameras keep it in view, so it is still there when the agents turn back.
World models that see only through the players' eyes forget what happens out of view.
OverSeer adds a few fixed cameras and generates every view together, so the world stays consistent over time and across players.
📄 arxiv.org/abs/2610.28554
🌐 runjiali-rl.github.io/OverSe…
🧵👇
Introducing Code Video Model 🎬
Amazing work led by @raywzy1 and the team!
Agentic coding as the engine for controllable video generation.
Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code.
A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance.
From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code.
Watch the demo and explore more visual results 👇
Project page: code-video-model.github.io/
Paper: code-video-model.github.io/t…
Code: github.com/code-video-model/…
Data: huggingface.co/datasets/code…
Youtube: youtube.com/watch?v=dTtiVkSR…
We share H3-World 🌍
The first to turn MiniMax-H3 itself into world model.
No new action module. We directly convert H3’s pretrained language understanding into world control.
Only 8K samples + 0.199% Trainable Params.
📄 huggingface.co/papers/2609.0…
💻 danzer1xxxxchan.github.io/H3…
We live in a multimodal world. We see, talk, act, and dream.
Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch?
Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition
So what’s the path forward?
Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects…)
We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.