←
David Hall2,034 posts
David Hall avatar

David Hall

@dlwh

Creator of Levanter and @MarinCommunity. MTS @ Open Athena. Previously Research Engineering @StanfordCRFM, co-founder at Semantic Machines ⟶ MSFT.

◷ Joined September 2007▧ 60 Photos and videos
David Hall avatar
David Hall@dlwh·16h

Unfortunately, we found substantial contamination from common pretraining benchmarks that our decontamination procedure missed. That affects which model comparisons we can fairly report. We’ve documented what happened and linked the investigation in the blog post.

David Hall avatar
David Hall@dlwh·16h

Open development is about more than releasing final weights and dumping a tech report on arXiv. People want to understand what happens during large-scale training, but usually can’t see it. These checkpoints are one way to help with that. (We'll do a tech report too.)

Percy Liang avatar
⟳ David Hall retweeted
Percy Liang@percyliang·Nov 10

Why was Marin 535B-A23B the size it was? Why 8 of 384 experts? Why GatedNorm, sconv, XSA? Why linear learning rate? All of these factors were driven by tradeoffs between timelines, compute efficiency, MoE router stability, etc. Read about the rationale of each design decision:

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

Post attachmentPost attachment
Larry Dial avatar
⟳ David Hall retweeted
Larry Dial@classiclarryd·Nov 9

Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.

Post attachmentPost attachment
David Hall avatar
David Hall@dlwh·Nov 5

It’s done! Marin’s Mochi 535B-A23B finished its main pretraining phase this morning. 18T tokens! Long context this weekend, SFT over the next week or so. Hoping to release checkpoints next week. Some EU paperwork first. mumtracker.exe.xyz/hero-run-…

Post attachment
Percy Liang avatar
⟳ David Hall retweeted
Percy Liang@percyliang·Nov 4

Marin 535B-A23B now on 65K contexts... Counting down...

Post attachment
Rohith Kuditipudi avatar
⟳ David Hall retweeted
Rohith Kuditipudi@rckpudi·Nov 3

The last paper from my PhD is finally out! Check out @suze_va's great thread and this talk I gave a few weeks back. Also a couple more takeaways in 🧵 below Zoom link: stanford.zoom.us/rec/share/f… Passcode: FZfpZ1p&

Can we forecast benchmark accuracy from models that score zero? We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵

Post attachment
Asuka avatar
⟳ David Hall retweeted
Asuka@HighFreqAsuka·Oct 28

I had a nice time at the Open Athena event on scaling Marin’s open LLM training. Thanks for the invite and good luck with the run!

Post attachment

Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. forge.coreweave.com/wandb/ma…

Post attachment
David Hall avatar
David Hall@dlwh·Oct 27

So, about that Uncheatable forecast. I used an estimate that wasn’t properly flop-controlled, and hadn’t checked its asymptote carefully enough. I should have checked it before posting. Easy fix. Details below.

Post attachment
David Hall avatar
David Hall@dlwh·Oct 26

Anyway, the lesson is we're not so much bad at forecasting as I'm bad at QA'ing my posts.

David Hall avatar
David Hall@dlwh·Oct 26

Of course this is retrospective. Hindsight etc. etc. That said, it was basically "do what we did for the Paloma fit". No 535b losses need have been consulted. As always, code and context are on GitHub github.com/marin-community/m…