
Very interesting! A new benchmark? 🧐
Ran a Writing Weights Index from today’s Weight-Space Programming post. Open-source models struggle on this task.

PhD Student @ Tsinghua University | Researching LLM

Very interesting! A new benchmark? 🧐
Ran a Writing Weights Index from today’s Weight-Space Programming post. Open-source models struggle on this task.

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 arxiv.org/abs/2608.27370 Kaiyuan-2B: PCMind-2.1-Kaiyuan-2B Technical Report arxiv.org/abs/2512.07612

From LR scaling laws to data curricula, we've been exploring better recipes for LLM pretraining. Many of these ideas have made their way into Puro-2B and Kaiyuan-2B, our fully open, efficient, low-cost pretraining projects. More to explore, and more to come!
see our series for the training recipes in pretraining! a scaling law that maps LR schedules to loss curves (multi power law) arxiv.org/abs/2503.12811 then studied how data reuse changes scaling across epochs arxiv.org/abs/2511.13421 how LR decay can weaken the benefits of data curricula. (CMA) arxiv.org/html/2511.18903v1

Learning rate affects how much an LLM learns from each data sample! Yet we often introduce high-value data during mid-training, as the LR decays. When we introduce that data matters! Check out our new work: Data Schedules and Learning Rate Schedules Should Be Designed Together!
(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule! At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better. We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end. See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !

Which optimizer is best? It turns out the answer can change with batch size! We perform a comprehensive sweep and find the best BSZ scaling rule depends on settings and can't be derived from early theories like SDE or convex bound!

Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched? We introduce SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. The answer is yes. And the advantage grows with scale. 🧵 1/8 Paper: arxiv.org/abs/2609.01343

Excited to introduce Puro-2B, along with its fully open recipe that lets you train a Qwen2-1.5B-level LLM on RTX 5090 GPUs for less than $5090! As someone who has always been curious about why LLMs work so well, one of the biggest challenges today is that we not only lack a good theory, but are also losing the ability to run controlled training experiments at scale, systematically vary key factors, and gather the empirical evidence needed to understand how LLMs learn from data. Training LLMs has simply become too expensive. There are now a few great open or almost open training recipes for models with billions of parameters, but reproducing them can still cost hundreds of thousands of dollars. As this makes many basic questions about LLMs inaccessible to academic labs, we really hope to bring this cost down far enough that academic labs can train and study genuinely capable models from scratch. Puro is a full-stack engineering effort to reduce training cost, which mixes engineering and research across data, optimization, numerical precision, hardware, and training strategy. It also builds on some of our own previous research: 📈 Curriculum Model Averaging (CMA): data curriculum designs have to work together with moderate learning rate decay and model averaging (arxiv.org/abs/2511.18903) ◉ Hyperball / MuonH: an optimizer wrapper that constrains weight and update norms, and directly controls the effective (angular) step sizes for stable and efficient training (arxiv.org/abs/2606.16899) There is still a long way to go, and we would be very happy to hear your feedback and ideas!
How much would it cost you to pretrain a 2B LLM from scratch? $1M? $100K? 🚀Announcing Puro-2B, with a fully open recipe. You can train a model that matches Qwen2-1.5B on RTX 5090s for less than $5090! $4.4K → beats Qwen2-1.5B $6.9K → approaches Qwen2.5-1.5B Check here👇

Puro-2B was a team effort 🙌 Fantastic work with Jiarui Cui, Yaorui Yin, Shengqi Chen (@Harry_Chen_ ), and many teammates, with guidance from Kaifeng Lyu (@vfleaking) and Wenguang Chen. We sincerely thank Yanfu Investments for providing GPU computing resources to support Puro.

Prior work behind Puro: 📚 Kaiyuan: arxiv.org/abs/2512.07612 🔌 RTX 5090 P2P: harrychen.xyz/2026/03/22/ena… 🌐 RTX 5090 GDR: harrychen.xyz/2026/05/20/ena… ◉ Hyperball: arxiv.org/abs/2606.16899 📐 Multi-Power Law: arxiv.org/abs/2503.12811 📈 CMA: arxiv.org/abs/2511.18903

Beyond cost, open pretraining enables us to ask a deeper question: Do pretraining choices still matter after SFT? Case study: curriculum+model average vs. uniform pretraining, followed by the same SFT: +1.77/+2.02 pp on GSM8K across two scales. Pretraining gains survive SFT.

A scaling law for scaling down 📉 The Puro Cost Scaling Law asks: How much capability can your budget buy? ✅ ~$4.4K crosses Qwen2-1.5B At the $6.9K endpoint: ↗ curriculum ≈ $11.4K uniform (1.65×) ⭐ full Puro recipe ≈ $16.6K uniform (2.40×)

No silver bullet. Our ablations show gains across the stack: • RTX 5090s — 2.77× BF16 compute/$ • blockwise FP8 — 1.34× matched-quality speedup • MuonH (Muon + Hyperball) — 1.19× • curriculum model averaging — 2.40× Each piece helps. Together, they make Puro possible.

So what made this possible? Puro co-designs the full pipeline: 📚 publicly accessible sources + proxy-guided selection → ⚡ RTX 5090s + blockwise FP8 → ◉ MuonH + effective-LR design → 📈 curriculum + checkpoint averaging → ⭐ Puro-2B From data to model 👇

Then why RTX 5090s? Under our pricing assumptions, RTX 5090 stands out in peak BF16 compute/$ (EFLOP/USD): 🥇 RTX 5090: 2.43 🥈 RTX PRO 6000: 0.96 🥉 H200: 0.89 A100: 0.63 That’s surprisingly strong economics for a consumer GPU! But hardware is only one piece of magic🧩

Back to the headline numbers: what do the reported $4.4K and $6.9K costs include? ✅ $4.37K → trained on 919B tokens using 14,262 GPU-hours ✅ $6.89K → trained on 1.4T tokens using 22,514 GPU-hours These are rental-equivalent GPU costs—not total R&D spend.

Puro-2B is open beyond the weights: ✅ technical report ✅ final + intermediate checkpoints ✅ training code + configs ✅ data-processing framework ✅ datasets + manifests 📄 Paper: arxiv.org/abs/2608.27370 💻 Code: github.com/thu-pacman/Puro-M… 🤗 Assets: huggingface.co/collections/t…