←
levi979 posts
levi avatar

levi

@levidiamode

365 days of GPU programming ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░░░░░░ 314/365

◷ Joined June 2025▧ 414 Photos and videos
levi avatar
levi@levidiamode·Mar 27

Day 83/365 of GPU Programming Looking at DeepSeek's Multi-Head Latent Attention today. The last part of the AMD challenge series is to optimize an MLA decode kernel for MI355X where the absorbed Q and compressed KV cache are given and your task is to do the attention computation. A resource that really helped internalize what MLA does was @rasbt's incredible visual guide to attention variants in LLMs (luckily he posted that last week!), which covers everything from MHA to GQA to MLA to SWA, et cetera. If there's one place to get a visual intuition for recent attention mechanisms, it's this blog post. @jbhuang0604's video on MQA, GQA,MLA and DSA was the best conceptual intro I found on the topic and progressively builds up the ideas from first principles. The Welch Labs analysis of MLA is a great watch as well. Beautiful visualization of the changes DeepSeek made for MLA. Tried out a few kernels once I had a basic understanding of MLA and I think I'm slowly getting more comfortable with at least analyzing kernels.

Post attachmentPost attachmentPost attachmentPost attachment

Day 82/365 of GPU Programming Taking a closer look at Mixture of Experts today, so I can write better MoE kernels. Specifically, to optimize an MXFP4 MoE fused kernel for the GPU Mode challenge. I haven't had much prior exposure to MoEs, so lots of new concepts I learned today. Luckily I found the best intro to MoEs thanks to @MaartenGr visual overview of the topic. I then watched @tatsu_hashimoto's amazing Stanford CS336 lecture on MoEs, which added deeper context around why MoEs are gaining popularity, FLOPs, OLMoE, infra complexity, routing functions (mindblown this works so well...), expert sizes, training objectives, top k routing and DeepSeek variations. Once I had a basic understanding I started playing around with the some AITER kernels but progress there is tbd. Also had a nice chat with @juscallmevyom (who was kind enough to reach out!) about the AMD kernels and the challenge of materialization overhead.

Post attachmentPost attachmentPost attachmentPost attachment
levi avatar
levi@levidiamode·16h

Day 317/365 of GPU Programming Just found out today the vLLM/inferact folks released a collection of megakernels for TPUs back in September which is a really interesting repo to scroll through. Also one fascinating thing about TPUs is how topologies affect kernel implementation (e.g. in this case, a 2×2×4 TPU chip topology)

Post attachmentPost attachmentPost attachment

Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.

Post attachmentPost attachmentPost attachmentPost attachment
levi avatar
levi@levidiamode·Nov 17

Day 317/365 of GPU Programming Spending some time going through the Pallas docs today. Docs are probably my favorite way to familiarize with a topic that doesn't have too many resources online. I like how the Jax team also has a section on creating a "snapshot of some of the earlier design decisions made" to understand how Pallas came to be.

Post attachmentPost attachment

Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.

Post attachmentPost attachmentPost attachmentPost attachment
Dylan Lim avatar
⟳ levi retweeted
Dylan Lim@dylan__lim·Nov 16

Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels

1:28

Two weeks ago we announced Together Compiler: agents that tune inference to your model, chip, and traffic, for up to 2.6x more throughput. Today, we’re sharing the journey behind it: the experiments that challenged our assumptions of megakernels, the surprising strengths of Torch Compile, and the lessons that shaped our development of a next-generation system for inference optimization. Shout out to @dylan__lim @reyna_abhyankar @simran_s_arora and the team! Read more: together.ai/blog/road-to-tog…

Post attachment
Ayaka Mikazuki avatar
⟳ levi retweeted
Ayaka Mikazuki@ayaka14732·Nov 15

I have a counterexample on TPU v7x. In a matmul kernel, issuing matmuls in bursts triggers power-management throttling, which in turn stalls weight delivery. Spacing out the matmuls yields a 25.9% speedup over an issue-when-ready greedy scheduler and 26.4% over vanilla Pallas.

Anyone who worked with wide instruction bundles / static scheduling on TPU, Hexagon, or AMD AI Engine — curious how close a greedy scheduler gets to optimal on your machines. I measured this on Elbrus (classic VLIW). Greedy is already optimal in 94% of cases, 0.51% cycle loss. Small neural net recovers most of the rest and never makes it worse. CLI that shows greedy vs exact optimum vs model on any .s file: github.com/smworklair/NEX-CL… @PatrickToulme @NormJouppi @adityazero_ @AminVahdat @JeffDean @Qualcomm @QCOMResearch @AMDembedded

levi avatar
levi@levidiamode·Nov 16

Day 316/365 of GPU Programming Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.

Post attachmentPost attachmentPost attachmentPost attachment

Day 315/365 of GPU Programming Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.

Post attachmentPost attachment
levi avatar
levi@levidiamode·Nov 14

Day 315/365 of GPU Programming Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.

Post attachmentPost attachment

Day 314/365 of GPU Programming Spending the weekend catching up on some lectures. Looks like the next section of the efficientml.ai course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.

Post attachmentPost attachment
levi avatar
levi@levidiamode·Nov 13

Day 314/365 of GPU Programming Spending the weekend catching up on some lectures. Looks like the next section of the efficientml.ai course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.

Post attachmentPost attachment

Day 313/365 of GPU Programming Reading up on some of the great posts on the LMSys blog. Especially interested in their recent megakernel article which delineates why SGLang has been moving towards larger GPU kernels. In it, they also describe what their kernels will evolve to (with a nod to the flappy folks) with a virtual machine “that takes control of the GPU and schedules different workloads across it”.

Post attachmentPost attachment
levi avatar
levi@levidiamode·Nov 12

Day 313/365 of GPU Programming Reading up on some of the great posts on the LMSys blog. Especially interested in their recent megakernel article which delineates why SGLang has been moving towards larger GPU kernels. In it, they also describe what their kernels will evolve to (with a nod to the flappy folks) with a virtual machine “that takes control of the GPU and schedules different workloads across it”.

Post attachmentPost attachment

Day 312/365 of GPU Programming So far really enjoying the zero to sglang course by @YichiZ03 which tries to take you from the basics of SGL by building your own "mini SGL" all the way to making your first contribution to their repo. Think this is a great playbook for OS projects in general to find and train future contributors. Wouldn't be surprised to see more projects come out with similar courses.

Post attachment