←
Aurko Roy142 posts
Aurko Roy avatar

Aurko Roy

@aurko79

Math & computer science | @AIatMeta (2025-2025) | @GoogleDeepmind (2023-2025) | @GoogleAI (Brain) (2017-2023) | CS PhD @Georgiatech | CS @IITKanpur

◷ Joined February 2025▧ 6 Photos and videos
Aurko Roy avatar
Aurko Roy@aurko79·4 Jul 2025

Excited to share what I worked on during my time at Meta. - We introduce a Triton-accelerated Transformer with *2-simplicial attention*—a tri-linear generalization of dot-product attention - We show how to adapt RoPE to tri-linear forms - We show 2-simplicial attention scales better under token constraints than dot product attention It was fun collaborating with amazing folks including @dvsaisurya @_arohan_ and others

Post attachment
Aurko Roy avatar
Aurko Roy@aurko79·Nov 3

Training across compute islands is going to become a very real constraint for a lot of labs as clusters stay fragmented and power/land get scarce. History shows scarcity is often the mother of the best algorithmic ideas — and some of the ideas explored here look genuinely promising!

Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️ coreauto.com/hitchhikers-gui…

Aurko Roy avatar
Aurko Roy@aurko79·Nov 2

Very cool work on content based sparse attention - look forward to trying it out!

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context. Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster. With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington

0:12
elie avatar
⟳ Aurko Roy retweeted
elie@eliebakouch·Oct 15

wow, amazing technical blog post about microsoft's new 3T pre-training with very rare and clear insights on how to train models of this size first let's get an overview of the shape of this model, it's a sparser model than Kimi K3 (80B active vs 104B), same sliding window hybrid pattern, with this time more kv heads for the sliding window layers (a bit different from stepfun and poolside that add query head). they added a shared expert and also removed the interleaved MoE/Dense pattern of MAI 1 the scaling ladder is very similar to MAI 1 with the same mechanism, each rung depends only on the number of layers. they mention that one of the key improvements is how they handle the smaller rung data distribution where they make sure to have the same repetition as the final corpus. they already did that for MAI-1 but only for data ablation iirc their prediction is almost identical to the final run loss, very impressive they compare the model with different base models such as deepseek v4 pro and MAI 1, tho in the case of deepseek v4 pro i'd say it's quite hard to distinguish how much the data mixture "overfitting" to the scaling ladder has an impact here (they mention it's both), need to always be a bit careful about EG metrics for data in general they have VERY NICE scaling laws for weight decay and learning rate, for the learning rate they express it as a function of tokens per parameter (N/D) and steps (hence depending on the batch size). they also do the same for dropout where it's a function of active params only, they also find the usual sqrt(k) scaling for learning rate and batch size (other scaling laws often also converge to that) a bunch of other stability stuff, they use Muon this time and activation gradient clipping, the efficiency gain here is actually pretty crazy at larger scale. they also increase the training horizon to ~40T text (20T unique) + a new 10T (4.5T unique) multimodal since the model is natively multimodal imo this is a safe but very clean scaling! very exciting for the tech report and MAI-base-3, congrats!!

Post attachment

We now have state-of-the-art pre-trained base models in the 3T-parameter weight class, built entirely in-house at MAI. Post-training results coming soon -- stay tuned! Lots of details in our technical blog post. microsoft.ai/news/frontier-p…

Lisan al Gaib avatar
⟳ Aurko Roy retweeted
Lisan al Gaib@scaling01·Sep 28

eventually they will all come around to the fact that we must computemaxx full attention or even more, like 2-simplical attention (but probably still hybrids) and of course loopmaxxing

Post attachment

Could long-context architectures “find all contradictions” in a science literature? Not yet! 🧵 We study a new class of "high-complexity” tasks whose difficulty scales quadratically with corpus size (as opposed to linearly), reversing common LCLM decisions! (block-sparse attention, hybrid models…)

Post attachment
rohan anil avatar
⟳ Aurko Roy retweeted
rohan anil@_arohan_·Sep 29

Super cool results on embedding table on Nanogpt wanted to point out ngrammer paper had a similar result and use of AdaGrad. Its really a free win.

Post attachmentPost attachment
Aurko Roy avatar
Aurko Roy@aurko79·Sep 29

Super cool results with some clever sharding and sparse updates to the 65B n-gram table obliterating the previous nanoGPT record!

New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…

Aurko Roy avatar
Aurko Roy@aurko79·Sep 11

The real innovation behind Moonshot's breakthrough was never Kimi Delta Attention (KDA)—it was Kimi Distillation Attack (KDA).

We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop. Read the report: anthropic.com/threat-intelli…

Aurko Roy avatar
Aurko Roy@aurko79·Aug 26

"The model looks up what it already knows for FREE and spends its compute purely on thinking. " This is the right intuition for why N-gram embeddings help.

Qwen3.8-Flash-Next Drops TOMORROW And I'm Deploying It LIVE 🚨 The architecture is wild: 125B brain, only 6B active per token, PLUS a 51B lookup table of pre-baked phrases. It's not more brain, it's flashcards 🧠⚡️ The model looks up what it already knows for FREE and spends its compute purely on thinking. Same quality as their flagship at 1/9th the training cost. Their last model this size ran 10B active. This runs 6. That's the speed unlock 📈 Tomorrow 11 AM I'm going live to crack this on the DGX Sparks. FP8 on TP2 first, then the community quant hunt begins. Can we fit it on ONE Spark? Can it run on four 3090s? We find out together 🔥 Stay tuned. You already know the numbers get posted here first 🎯

5:57
Aurko Roy avatar
Aurko Roy@aurko79·Aug 25

51B N-gram table is massive!

Qwen 3.8 Flash Next is releasing Tomorrow. 125B paramters +51B N-gram and 6B active. Its based on the next generation Qwen 4 architecture. Qwen 4 is coming

Post attachment
Aurko Roy avatar
Aurko Roy@aurko79·Jun 12

Nice to see MiniMax and others pushing content based sparse attention forward. Back in 2020 the field was exploding with several efficient attention variants: - Local/Sliding window attention - Fixed sparsity patterns - Strided attention - Recurrence ( T-XL, compressive transformers) - Chunked attention - Linear attention and SSMs The right prior which has stood the test of time is a combination of: a) local attention for modeling local context and b) a top-k content based sparsity to route queries to the most relevant keys (blocks of keys). What has changed in the last 6 years is better hardware-algorithm codesign.

Post attachment

Hey everyone — our high-performance MSA kernel library is now open-source. The M3 weights are expected to drop this Friday. Thanks for waiting! Github: github.com/MiniMax-AI/MSA Paper:github.com/MiniMax-AI/MSA/bl…

Post attachment
Aurko Roy avatar
Aurko Roy@aurko79·Jun 10

As someone who watched Rohan train LLMs with shampoo and brainstorm ideas for making it cheaper/more scalable back in 2021-2022 (low-rank shampoo for very large weights etc), I think it's fair to say that Shampoo was certainly a step change over Adam. That said Muon also introduced several innovations and it may be an over-simplification to say that it's just Shampoo with b2=0. - For one the use of Nesterov momentum - Second the observation that computing inverse roots is inherently more numerically unstable than orthogonalization - Clever use of Newton-Schulz which makes it more GPU friendly than eigh or QR or coupled newton - Scaling the update rms norm to match 0.2 (Adam update rms in the limit) which made grafting with Adam unnecessary I think both optimizers are awesome and I learned a lot from studying both of them. Progress doesn't happen in a vacuum, we all stand on the shoulders of giants.

not sure how you can say the first major innovation since adam was muon and not shampoo. although I'll admit that muon is much simpler to implement in practice! unless you are saying muon is a special case of shampoo with b2=0?

rohan anil avatar
⟳ Aurko Roy retweeted
rohan anil@_arohan_·Jun 9

Here you go, sir. Muon is a good optimizer. I think Keller's attempt at implementing it is great -- this primarily helped me look at why his effort produced much worse looking curve than what I could get. This ended up being a nerdsnipe into hyper parameters, grafting, and eigh calls in the distributed shampoo package from Meta in my car ride home. The main delta's are below:

Post attachment

I've added two optimizers to the public benchmark: (1) Shampoo (with its original 1/4 power). (2) Spectral descent, which is equivalent to both Muon(mu=0) and Shampoo(b1=b2=0). Result: Shampoo falls halfway between Muon & Adam; Spectral descent is ~2x slower. Thread below 1/6

Post attachment