
tuning into @dlwh share the behind the scenes of @MarinCommunity ! any questions we should ask him?

Creator of Levanter and @MarinCommunity. MTS @ Open Athena. Previously Research Engineering @StanfordCRFM, co-founder at Semantic Machines ⟶ MSFT.

tuning into @dlwh share the behind the scenes of @MarinCommunity ! any questions we should ask him?

These are intermediate pre-training checkpoints: no instruction tuning or safety post-training. We'll have a lot more to say about this run, but in the meantime, we’d love to hear what you find. huggingface.co/open-athena/M… Blog: openathena.ai/blog/marin-moc…

Unfortunately, we found substantial contamination from common pretraining benchmarks that our decontamination procedure missed. That affects which model comparisons we can fairly report. We’ve documented what happened and linked the investigation in the blog post.

Open development is about more than releasing final weights and dumping a tech report on arXiv. People want to understand what happens during large-scale training, but usually can’t see it. These checkpoints are one way to help with that. (We'll do a tech report too.)

We’re releasing all 57 intermediate checkpoints from Marin “Mochi” 535B-A23B pretraining, including 10 from long-context extension. 18.2T tokens, about 61 TB of weights. huggingface.co/open-athena/M… Blog: openathena.ai/blog/marin-moc…

We hosted @dlwh for a fantastic talk about Marin pre-training about two weeks ago youtube.com/watch?v=hlbj9dHo… If you want to read more beyond the talk, they just posted about their pre-training design decisions openathena.ai/blog/scaling-m…

Why was Marin 535B-A23B the size it was? Why 8 of 384 experts? Why GatedNorm, sconv, XSA? Why linear learning rate? All of these factors were driven by tradeoffs between timelines, compute efficiency, MoE router stability, etc. Read about the rationale of each design decision:
Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

BTW @WilliamBarrHeld has a great new blog post up from last week about what got us here. openathena.ai/blog/scaling-m…

It’s done! Marin’s Mochi 535B-A23B finished its main pretraining phase this morning. 18T tokens! Long context this weekend, SFT over the next week or so. Hoping to release checkpoints next week. Some EU paperwork first. mumtracker.exe.xyz/hero-run-…

Marin 535B-A23B now on 65K contexts... Counting down...

The last paper from my PhD is finally out! Check out @suze_va's great thread and this talk I gave a few weeks back. Also a couple more takeaways in 🧵 below Zoom link: stanford.zoom.us/rec/share/f… Passcode: FZfpZ1p&
Can we forecast benchmark accuracy from models that score zero? We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵

I had a nice time at the Open Athena event on scaling Marin’s open LLM training. Thanks for the invite and good luck with the run!
Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. forge.coreweave.com/wandb/ma…

So, about that Uncheatable forecast. I used an estimate that wasn’t properly flop-controlled, and hadn’t checked its asymptote carefully enough. I should have checked it before posting. Easy fix. Details below.

Anyway, the lesson is we're not so much bad at forecasting as I'm bad at QA'ing my posts.

Of course this is retrospective. Hindsight etc. etc. That said, it was basically "do what we did for the Paloma fit". No 535b losses need have been consulted. As always, code and context are on GitHub github.com/marin-community/m…