←
Junlin Han349 posts
Junlin Han avatar

Junlin Han

@han_junlin

MTS @ @amilabs | Pre-training & core capabilities. Ex Meta FAIR. PhD Oxford. Opinions are my own.

◷ Joined July 2020▧ 28 Photos and videos
Junlin Han avatar
Junlin Han@han_junlin·Aug 6

We live in a multimodal world. We see, talk, act, and dream. Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch? Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition So what’s the path forward? Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects…) We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

Post attachment
Junlin Han avatar
Junlin Han@han_junlin·Oct 31

I joined @amilabs ~a month ago as an MTS, working on world model pre-training and capability emergence. There has never been a better time to build AI for the real world with a scaling mindset. Thrilled to join forces with friends old and new to build the next frontier🚀!

Tianzhe Chu @ Neurips (Sydney) avatar
⟳ Junlin Han retweeted
Tianzhe Chu @ Neurips (Sydney)@TianzheC·Oct 27

Introducing Nestor! In standard MoEs, tokens choose their FFN experts. What if they could choose their attention modules, too? 🤔 We introduce a transformer with nested sparsity for scalable pretraining and continual learning.

1:24
Runjia Li avatar
⟳ Junlin Han retweeted
Runjia Li@RunjiaLi·Oct 26

World models that see only through the players' eyes forget what happens out of view. OverSeer adds a few fixed cameras and generates every view together, so the world stays consistent over time and across players. 📄 arxiv.org/abs/2610.28554 🌐 runjiali-rl.github.io/OverSe… 🧵👇

1:24
Chunyi Sun avatar
⟳ Junlin Han retweeted
Chunyi Sun@ChunyiSun99·Sep 14

🎉 AnimalLift has been accepted to SIGGRAPH Asia 2026! 🐶🐱🐻🐯🦊 From a single image, AnimalLift reconstructs an animation-ready 3D animal with canonical geometry, UV texture, and explicit strand-based fur. 📄 arxiv.org/abs/2609.09513
🤗 huggingface.co/datasets/Chun…

1:30
Junlin Han avatar
Junlin Han@han_junlin·Sep 10

Computer vision is officially 3D only. 7 out of the 10 candidates are 3D. Multimodal foundation models, generative scaling, representation learning? Irrelevant to intelligence. The universe is made of point clouds, meshes, and Gaussians. (Nothing against the authors btw)

Post attachment
Jihan Yang avatar
⟳ Junlin Han retweeted
Jihan Yang@jihanyang13·Aug 31

New positions open! We’re looking for full-time researchers and interns with backgrounds in geometry and 3D vision to join our team at AMI Labs. The roles can be based in NYC, Paris, Montreal, or Singapore. If you’re interested, apply through the link below: jobs.ashbyhq.com/ami/0d7332d…

Xidulu avatar
⟳ Junlin Han retweeted
Xidulu@xidulu·Aug 11

1/ Sharing a new, interesting project I did during my internship at Microsoft AI Frontiers w/ @JohnCLangford TL;DR: At decoding time, we feed **previous hidden state** into the input together with token embedding, and it boosts performance for free. arxiv.org/abs/2608.08888

Junlin Han avatar
Junlin Han@han_junlin·Aug 6

[5/5] This is likely the final paper of my PhD. When I began my PhD around the release of GPT-4V (Oct 2023), the AI landscape was very different. My entire doctoral journey has been a progression from computer vision research to cross-modality interaction, and ultimately to native multimodality. I hope our exploration serves as a small step towards advancing multimodal intelligence. I still believe vision is a fundamental component of intelligence, and future models should pursue natively multimodal co-evolution, even if it's harder. Please visit our webpage for more details: junlinhan.github.io/projects… Joint work with @TongPetersb , @DavidJFan , @MinghaoChen23, @philiptorr , @filippos_kok, and @ml_perception .

Junlin Han avatar
Junlin Han@han_junlin·Aug 6

[4/5] Recipe: Translating physical mechanics into a practical scaling recipe: Data recipe: Allocating as little as 5% of the token budget to visual generation achieves competitive generative modeling while preserving strong language & understanding capabilities. Scale Verification: Validated on multiple 13.5B MoE models trained across 2T tokens , proving that early fusion + asymmetric data blend + MoE scales efficiently.

Post attachment
Junlin Han avatar
Junlin Han@han_junlin·Aug 6

[3/5] Early Unification: Timing matters. Early unification with joint training allows modalities to better co-evolve. Simultaneous early fusion significantly outperforms late alignment or sequential training. Delayed integration causes "Vision Laziness": Pretraining on text first creates strong language priors, causing the model to perceive and learn fewer visual signals.

Post attachmentPost attachmentPost attachment
Junlin Han avatar
Junlin Han@han_junlin·Aug 6

[2/5] Modality Synergy: Joint training isn't automatically synergistic; interaction depends heavily on task complexity and architecture. Key findings across our experimental axes: Task complexity: Simple tasks foster cross-modal synergy, while complex tasks trigger competition. Architecture sharing: Share Attention & Norms to drive cross-modal interaction and synergy; Decouple FFNs to eliminate capacity bottlenecks and parameter competition. Encoder generalization: These architectural guidelines (and knowledge flow results) consistently hold across diverse vision representations (continuous vs. discrete, pixel-level vs. semantic).

Post attachmentPost attachmentPost attachmentPost attachment
Junlin Han avatar
Junlin Han@han_junlin·Aug 6

[1/5] Knowledge flow: Knowledge flow is highly asymmetric and concept-dependent. We demonstrate this in both real-world data and controlled synthetic environments: Language is a universal booster for all visual tasks. Understanding is a strong prior for generation, significantly boosting visual creation. Generation acts as a latent prior. it doesn’t yield instant zero-shot gains for understanding, but sharply accelerates low-level concept acquisition.

Post attachmentPost attachmentPost attachment
Muzi Tao avatar
⟳ Junlin Han retweeted
Muzi Tao@tao_muzi·Jul 10

🎉 Our paper, “Asymmetric Idiosyncrasies in Multimodal Models,” is accepted to @COLM_conf 2026! You can tell WHICH model wrote a caption with 99.53% accuracy. Render it into an image, and attribution collapses to 49.85%. Where does the fingerprint go? 🧵 1/5

Post attachment