←
Marin Community14 posts
Marin Community avatar

Marin Community

@MarinCommunity

Open research for frontier AI. Building models, datasets, infrastructure, and experiments in public. Toward an Open Frontier

◷ Joined September 2026
Larry Dial avatar
⟳ Marin Community retweeted
Larry Dial@classiclarryd·Nov 9

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

Post attachmentPost attachment
David Hall avatar
⟳ Marin Community retweeted
David Hall@dlwh·Oct 15

I've been thinking about where the data for the next generation of open models will come from. The internet gave us a remarkable commons for pretraining. Posttraining is different: it consumes tasks, not documents. And that commons doesn't exist. Not yet. openathena.ai/blog/there-is-…

Percy Liang avatar
⟳ Marin Community retweeted
Percy Liang@percyliang·Oct 6

🚢Marin 535B-A23B training has crossed the halfway mark! Some observations and learnings so far:

Post attachment
Will Held avatar
⟳ Marin Community retweeted
Will Held@WilliamBarrHeld·Oct 2

If you're keeping tally of our blogs @ravwojdyla enabled: - ~All the infra that enabled our data scale - A kernel that let us ~2x our parameter count while maintaining higher MFU 🐐

43 days ago Marin kicked off the largest live-streamed pretraining run in history: a 535B MoE trained on 18T tokens. A scaling ladder was used to forecast eval loss along the entire trajectory, and the forecast was publicly pre-registered. Its halfway along, and despite being a 300x extrapolation, the run is within 0.3% of the forecast. The run was originally planned to be 360B parameters. Until @ravwojdyla found a hardware implementation with custom kernels to grow to 535B parameters while also speeding up tokens per second. Learn about how he did it on the Open Athena blog: openathena.ai/blog/expert-pa….

Post attachmentPost attachment
Rafal Wojdyla avatar
⟳ Marin Community retweeted
Rafal Wojdyla@ravwojdyla·Oct 1

Marin’s 535B-parameter hero run was originally planned as a 360B model. In openathena.ai/blog/expert-pa… we explain how expert parallelism made room for a roughly 50% larger model, with approximately 23B active parameters per token - and the tradeoffs involved.

Will Held avatar
⟳ Marin Community retweeted
Will Held@WilliamBarrHeld·Sep 23

There’s an enormous amount of open training data on Hugging Face. What does it take to make it work together 🤗? For Marin’s 535B run, we built on 25T tokens from 152 datasets with licenses permitting training. Here’s the work between downloading those and training a model 🧵

Post attachment
David Hall avatar
⟳ Marin Community retweeted
David Hall@dlwh·Sep 3

It’s been a few weeks since Marin kicked off our 535B/A23B MoE hero run. So far it’s gone almost boringly well. It’s also been just over a year since Marin joined Open Athena. I wrote up some thoughts about how far we’ve come, and how we got there. openathena.ai/blog/marin-535…

Post attachment
Larry Dial avatar
⟳ Marin Community retweeted
Larry Dial@classiclarryd·Jun 3

Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions. openathena.ai/blog/pretraini…

Will Held avatar
⟳ Marin Community retweeted
Will Held@WilliamBarrHeld·May 11

To train better open models, we need predictable scaling. Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error. Getting there took some work 🧵

0:10
Percy Liang avatar
⟳ Marin Community retweeted
Percy Liang@percyliang·Aug 21

🚢 Marin 535B-A23B started training this week! As usual, the whole process is open. Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow. Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.

Post attachment
David Hall avatar
⟳ Marin Community retweeted
David Hall@dlwh·26 Jun 2025

So about a month ago, Percy posted a version of this plot of our Marin 32B pretraining run. We got a lot of feedback, both public and private, that the spikes were bad. (This is a thread about how we fixed the spikes. Bear with me. )

Post attachment

Marin 32B training crossed 1.5 trillion tokens today...

Post attachment
Percy Liang avatar
⟳ Marin Community retweeted
Percy Liang@percyliang·19 May 2025

What would truly open-source AI look like? Not just open weights, open code/data, but *open development*, where the entire research and development process is public *and* anyone can contribute. We built Marin, an open lab, to fulfill this vision:

Post attachment