←
Sebastian Raschka20,425 posts
Sebastian Raschka avatar

Sebastian Raschka

@rasbt

ML/AI research engineer. Ex stats professor. Author of "Build a Large Language Model From Scratch" (amzn.to/4fqvn0D) & reasoning (mng.bz/lZ5B)

◷ Joined October 2012▧ 2,212 Photos and videos
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Sep 9

I put together a mega write-up on GPT-6 Astra & looped transformers. How looped transformers / recurrent depth works, cost-tradeoffs, whether it hides reasoning traces, with lots of figures and a tour of recent looped transformer research.

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·18m

Comparing new LLM releases via benchmarks is now a bit like using IQ tests for hiring decisions. Sure, benchmarks are useful baselines and so on, but beyond that, I don't know how indicative they are of our individual tasks and needs. (Yes, also looking at that release yesterday)

Sebastian Raschka avatar
Sebastian Raschka@rasbt·23h

Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models

Post attachment

Trick #3: Merge, merge, merge. In pretraining, we skipped LR decay for a running weight average. Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains). Merging was also an expedient stand-in for MOPD in our RLVR recipe.

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 16

The open-source LLM ecosystem could really benefit from an LLM building-block API. We have so many LLM components across different models now. Whether you are using PyTorch or C++ and CUDA code directly, there is a major switching cost when you want to run experiments or ablation studies, say swapping GQA with MLA, dense FFNs with MoE, attention layers with Mamba2 or Gated DeltaNet, or standard residual connections with attention residuals. Currently, it’s very messy even with LLM-assisted coding. It could maybe be on top of PyTorch or CUDA directly, but it needs to be carefully engineered and maintained (versus quickly vibe-coded on a weekend) and efficient enough that people trust it to use in production (which is really a requirement to move beyond toy examples). Maybe some food for thought for a big, well-funded lab :) (There are some efforts that go into that direction, like torchtitan, torchtune, Megatron Core, OLMo-core, but they are not quite that level of abstraction, I'd say. And, as always, there’s the chicken-and-egg problem: it needs to be popular to get widely adopted, and it needs to get widely adopted to become popular.)

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 15

(Finally) wrote up "Understanding Positional Embeddings in Modern LLMs." A RoPE from-scratch tutorial missing from my LLM building blocks collection + what recent architectures do (NoPE, partial RoPE, hybrids). Prob not the flashiest topic, but essential for understanding LLMs

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 14

Inkling 2 Small took a while... but it's more interesting than I initially thought! Using attention residuals (like Kimi K3), but without the linear-attention hybrid. Food for discussion for the end-of-the-year architecture trends analysis!

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 14

Seb's Law, co-signed by another Seb

Seb's Law: The widespread, transformative societal impact of AI is always approximately two years away, regardless of the current year.

Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 13

Wow that escalated quickly. Was gone for a few weeks and now pretty much everyone launched their own decision model copycats: - Sep 23: Together AI, Tev1-4B - Sep 24: Alibaba Cloud, decision-model - Sep 29: Liquid AI, d1 - Sep 30: Databricks, ai_decide - Oct 1: Amazon AWS, Strands Decider 2B - Oct 1: Cloudflare, Clef and Clef-flash (Clef-omni on Oct 9) - Oct 1: Perplexity, Decider (v1.3 on Nov 12) - Oct 6: OpenAI, Decisions API (GPT-6 Luna) - Oct 7: H2O.ai, H2O-Lightning-4B - Oct 7: Snowflake, Snowflake Decision - Oct 9: Microsoft, Microsoft-Decision-1 - Oct 15: IBM, Granite Switch 4.1 with Decision adapter - Oct 21: Baseten, Mammoth-1 and Elephant-1 - Nov 6: Salesforce, Koa Decisions - Nov 11: Google DeepMind, Gemini Decision 1 According to a Vercel report, 26% of requests are now routed through decision models 📈

Took a while, but here it is, a mega write-up on language models for text classification and, of course, Jev. It's basically a visual guide to RNNs, CNNs, transformers, and calibration, with hands-on experiments on accuracy and efficiency.

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 12

Hoping Nemotron keeps going. Thankfully, open weight LLMs have some alternatives. But their open data and full training recipes would be much harder to replace.

Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 12

LLM training is hard. Hard to be a jack of all trades.

im surprised there isnt more pretrain-postrain codesign i guess companies really do ship their org charts

Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 12

In reasoning model development, it's always been this interesting tension between training/optimizing chain-of-thoughts (CoTs) vs not touching them to preserve monitorability. CoTs are cool because we can check what the model is up to (e.g., whether it cheats or not) as long as it spells out its thought process. And we risk losing that if we explicitly optimize CoTs. Two caveats: 1) I.e., the catch of CoT and monitorability is that this only works if the written thinking reflects what the model is actually doing. 2) Also, when models become more capable, they spell out less in the CoT (like a cleverer math student needs less note-taking when solving problems during an exam). Nothing new on that front, but an interesting thread below on how to measure "harder to monitor", with some interesting results that confirm the existing intuitions: 1) no-CoT capability (meaning how much reasoning the model can do in a single forward pass without writing anything down) - Better pretraining (lower perplexity) raises this no-CoT capability - Architecture mostly doesn't matter 2) CoT controllability (meaning how well the model can change its written reasoning when asked to add, remove, or change parts of it) - RL alone raises controllability a lot - Controllability is passed on through distillation * Higher no-CoT capability and higher controllability both make monitoring harder. PS: Perhaps the good news is that it seems that process supervision wasn't the main culprit.

(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵

Post attachment
Brian avatar
⟳ Sebastian Raschka retweeted
Brian@brianrey_x·Nov 11

i've never learned as much as when building things from scratch. currently working through build a large language model by @rasbt, and it's one of the best ways i've found to actually understand how llms work. still struggling to visualize all those dimensions.

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 11

Hillclimbing: Make the model better (best intentions). Benchmarking: Measure how much better (best effort). Benchmaxxing: Make the numbers look better (best optics).

We gotta have a conversation about hill climbing vs benchmarking vs benchmaxxing

Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 10

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues. In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids. In the first half of 2026, the trend towards "some full attention" hybrids continued. And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore. The plot legend here means, - Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer. - Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention. - No full attention = Zero dense global layers. The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls. In particular, there are two distinct efficiency gains here: 1) Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.) 2) Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

Post attachment
Sebastian Raschka avatar
Sebastian Raschka@rasbt·Nov 3

What could possibly go wrong?

Post attachment

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…

Post attachment