←
Runjia Li62 posts
Runjia Li avatar

Runjia Li

@RunjiaLi

DPhil in Computer Vision, University of Oxford || Clarendon Scholar @Oxford_VGG @OxfordTVG || Experience with @Snap @AIatMeta

◷ Joined October 2022▧ 10 Photos and videos
Runjia Li avatar
Runjia Li@RunjiaLi·Nov 12

📄 Paper: arxiv.org/abs/2611.14420 🌐 Page: runjiali-rl.github.io/onlyac… 💻 Code + weights: github.com/runjiali-rl/onlya… With @MinghaoChen23 , @JunyuXieArthur , @philiptorr , Andrea Vedaldi and @JakabTomas (University of Oxford). See you at #NeurIPS2026!

Runjia Li avatar
Runjia Li@RunjiaLi·Nov 11

How? Contrastive learning on raw video, no labels: ➕ Positive: optical flow finds what moves; we cut it out and paste it into another scene. Same action, new scene → pull together. ➖ Hard negative: another moment of the same video. Same scene, new action → push apart.

0:07
Runjia Li avatar
Runjia Li@RunjiaLi·Nov 11

One still image, many actions: start from one anime-style robot picture and replay the arm motions of three robot demos (mug onto a plate, book onto a shelf, sweeping), just by swapping the clip. 🎨 And every action works in every world: 6 camera motions × 4 imagined worlds.

0:15
Runjia Li avatar
Runjia Li@RunjiaLi·Nov 11

Robots, too. Demonstrations from robot arms (UT Austin MUTEX, Berkeley UR5) drive the arm in the LIBERO simulator, and motions carry over between LIBERO and RLBench. 🤖 Large video: generated. Inset: the source clip.

0:06
Runjia Li avatar
Runjia Li@RunjiaLi·Nov 11

Give OnlyAct a source clip: a racing game, a robot demo, a shooter. It reads the action from each pair of frames, and a world model plays it out in a new scene, here from a single image. Forza → kart track 🏁 LIBERO → unseen MetaWorld arm 🤖 Valorant → lantern-lit alley

0:15
Yushi LAN avatar
⟳ Runjia Li retweeted
Yushi LAN@yushilan98·Oct 29

🎥Reshoot a video from a camera that was never there~ Excited to share #Holo4D, accepted to #neurips2026 ! Give it a video and a new camera path: Holo4D lifts the video into 3D tracks and uses them to steer a video model. The camera moves; what happened stays the same. 🧵

2:05
Runjia Li avatar
Runjia Li@RunjiaLi·Oct 26

It works in Minecraft, in Doom and on real multi-camera capture: places stay where they were when a player returns, and what one player does is seen by the other.

0:24
Runjia Li avatar
Runjia Li@RunjiaLi·Oct 26

Fixed cameras barely change, so our δ-VAE keeps each camera's first frame and codes only what changes: 8× fewer tokens per camera. The agents' views stay full frames. OverSeer then generates all views together, frame by frame, from the agents' actions and camera poses.

0:19
Runjia Li avatar
Runjia Li@RunjiaLi·Oct 26

A creeper explodes while both agents look away. A world model built on egocentric views alone loses the crater. OverSeer's fixed cameras keep it in view, so it is still there when the agents turn back.

0:17
Runjia Li avatar
Runjia Li@RunjiaLi·Oct 26

World models that see only through the players' eyes forget what happens out of view. OverSeer adds a few fixed cameras and generates every view together, so the world stays consistent over time and across players. 📄 arxiv.org/abs/2610.28554 🌐 runjiali-rl.github.io/OverSe… 🧵👇

1:24
Gordon (Guocheng) Qian avatar
⟳ Runjia Li retweeted
Gordon (Guocheng) Qian@guocheng_qian·Oct 12

Introducing Code Video Model 🎬 Amazing work led by @raywzy1 and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: code-video-model.github.io/ Paper: code-video-model.github.io/t… Code: github.com/code-video-model/… Data: huggingface.co/datasets/code… Youtube: youtube.com/watch?v=dTtiVkSR…

2:18
Xingyi Yang avatar
⟳ Runjia Li retweeted
Xingyi Yang@yxy2168·Sep 2

We share H3-World 🌍 The first to turn MiniMax-H3 itself into world model. No new action module. We directly convert H3’s pretrained language understanding into world control. Only 8K samples + 0.199% Trainable Params. 📄 huggingface.co/papers/2609.0… 💻 danzer1xxxxchan.github.io/H3…

0:49
Junlin Han avatar
⟳ Runjia Li retweeted
Junlin Han@han_junlin·Aug 6

We live in a multimodal world. We see, talk, act, and dream. Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch? Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition So what’s the path forward? Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects…) We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

Post attachment