←
Maximilian Böther893 posts
Maximilian Böther avatar

Maximilian Böther

@MaxiBoether

making data loaders go brr (deterministically) | mts @datologyai | prev. Ph.D. @SystemsGroupETH @anaklimovic

◷ Joined October 2012▧ 20 Photos and videos
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 7

At @datologyai we run thousands of data experiments. Two things keep biting us: - Same run on 16 vs 8 GPUs → different evals - Every pipeline change → re-materialize the dataset Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵

1:06
DatologyAI avatar
⟳ Maximilian Böther retweeted
DatologyAI@datologyai·Oct 29

When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees

Post attachment
Florian Gallwitz avatar
⟳ Maximilian Böther retweeted
Florian Gallwitz@FlorianGallwitz·Oct 27

Denken wir einen Moment an die Journalisten, die uns heute 3-SAT erklären müssen, für die das bisher ein Sender war.

Ari Morcos avatar
⟳ Maximilian Böther retweeted
Ari Morcos@arimorcos·Oct 16

Awesome to see Zephon, our recently open sourced data loader, integrated with @PyTorch TorchTitan! Check out their blog, and congrats to @MaxiBoether, @josh_wills, and @SGui. We're just getting started with Zephon!

Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count. In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus. The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count. ✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai) Read more: pytorch.org/blog/bringing-el…

Post attachment
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 16

Very happy that the Zephon team and Pytorch/Torchtitan are working together to make data loading more deterministic. Check out our example integration: github.com/datologyai/torcht… Check out our blog post together with the folks at Meta 👇

Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count. In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus. The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count. ✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai) Read more: pytorch.org/blog/bringing-el…

Post attachment
Matthew Leavitt avatar
⟳ Maximilian Böther retweeted
Matthew Leavitt@leavittron·Oct 10

MosaicML’s Streaming dataloader was one of the most underrated pieces of ML infrastructure ever built imo. It spoiled me. One of my goals when founding Datology was to take all the best things about Streaming and build its torchbearer. Because well-designed dataloaders enable entire domains of data experimentation like real-time curricula, arbitrary mixing schedules, and seamless transitions between pre- and post-training. And even better-designed dataloaders enable productionizing and scaling those experiments. I am so excited that we're finally releasing Zephon. It leveled up Datology and we think it'll level you up, too.

At @datologyai we run thousands of data experiments. Two things keep biting us: - Same run on 16 vs 8 GPUs → different evals - Every pipeline change → re-materialize the dataset Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵

1:06
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 9

A data loader is only as useful as the data it loads. This is why this week is so exciting: we're sharing our Curation Studio and how we use it to generate the best tokens (or raw strings to be tokenized and packed on-the-fly) possible. Check out our release announcement!

Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models. Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.

2:34
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 7

[8/] We originally built Zephon for our own data curation experiments. Getting its primitives right was tricky and a longer process; elastic determinism for stateful pipelines has subtle failure modes. We strive for a world in which anybody can build their own foundation model.

Post attachment
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 7

[7/] Researchers and engineers spend time repeatedly reinventing data loaders and it’s time for a modern, text- and multimodal data loader that embraces online processing while keeping the guarantees of offline processing.

Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 7

[6/] Everybody can bring their own setup and format. Zephon’s API is flexible, and you can do as many things online (or offline) as you want. We can load parquet, litdata, vortex, jsonl, and more: we don’t lock you in into anything.

Post attachment
Maximilian Böther avatar
Maximilian Böther@MaxiBoether·Oct 7

[5/] We do all of this while being fast: we separate stateful decision making and parallel processing, and provide a toolbox of useful LLM/VLM operators in this framework. We also are flexible in the execution backend: the future will be GIL-free, and Zephon is ready for it.

Post attachment