←
Albert Gu674 posts
Albert Gu avatar✓

Albert Gu ✓

@_albertgu

assistant prof @mldcmu. chief scientist @cartesia. leading the ssm revolution.

◷ Joined December 2018▧ 63 Photos and videos
Albert Gu avatar
Albert Gu@_albertgu·Oct 28

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

1:19
Albert Gu avatar
Albert Gu@_albertgu·Nov 8

i mean i'll take it (but really our research team is incredible)

Post attachment
Albert Gu avatar
Albert Gu@_albertgu·Nov 7

it's hilarious how many have taken the bitter lesson to mean just scaling up transformers. it's about scaling, but innovations in algorithmic primitives can lead to much more efficient scaling

Neehal Tumma avatar
⟳ Albert Gu retweeted
Neehal Tumma@ntumm120·Nov 5

We've spent years learning how to quantize attention. But what about the linear recurrences (e.g. Mamba-2, GDN and KDA) replacing it? It turns out we can rewrite their computation to reduce quantization error, while preserving the full-precision function. 🤯 Introducing LinQuant: a framework for low-precision recurrent prefill. 🧵 (1/7)

0:14
Thomas Wolf avatar
⟳ Albert Gu retweeted
Thomas Wolf@Thom_Wolf·Nov 5

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: huggingface.co/spaces/huggin…

2:42
Albert Gu avatar
Albert Gu@_albertgu·Nov 4

new open weights model from a dark horse

1:14
Albert Gu avatar
Albert Gu@_albertgu·Nov 4

in a constantly changing model landscape, faster adaptation is the biggest advantage

Albert Gu avatar
Albert Gu@_albertgu·Nov 4

byte-level models are better under multi epoching (e.g. training K times over 1/K of the data). another connection between learning from raw data <-> improved data efficiency

Post attachment
Albert Gu avatar
Albert Gu@_albertgu·Oct 31

this is an abuse of technology 😭

0:30
Albert Gu avatar
Albert Gu@_albertgu·Oct 30

Our opening animation and blog didn't fully show the interesting dynamics for languages like CJK that have multi-byte characters. In Chinese most characters take up 3 bytes, and when learning end-to-end from bytes it's not even trivial to get the model to chunk these into characters. Early in training this H-Net incorrectly sub-divides characters, but it eventually figures it out. Actually this was a prominent issue in the original H-Net - and even BPE doesn't get it right on rare characters. The blog has some examples!

0:13
1:19
Albert Gu avatar
Albert Gu@_albertgu·Oct 29

We build not just models but the whole stack - #1 on Speech Agent Arena!

1:05
Cartesia avatar
⟳ Albert Gu retweeted
Cartesia@cartesia·Oct 27

Cartesia Managed Agents is #1 on the @ArtificialAnlys Speech Agent Arena, outperforming 20+ speech-to-speech systems on tasks like appointment booking and ordering. We achieve frontier quality and naturalness by co-designing our speech models and harness in one vertically integrated stack.