
When building frontier multimodal models like Muse*, we keep coming back to how to design the vision encoder. It’s an architectural decision we have to make very early. This work finally gave us a chance to step back and study it systematically. Check our new paper!
Labs disagree on vision encoders for MLLMs! ⚔️ Gemma 4 12B scrapped the encoder. Kimi K3 dropped encoder pretraining. Inkling 2 Small revived the encoder Inkling killed. We trained 200+ models in a controlled study to settle it: ✅ Keep vision parameters, skip the pretraining. And they don't have to be an encoder! 🤔 Paper: arxiv.org/abs/2611.17334 🧵




























