←
GPU MODE274 posts
GPU MODE avatar

GPU MODE

@GPU_MODE

Your favorite GPU community

◷ Joined September 2024▧ 43 Photos and videos
Tanishq Mathew Abraham, Ph.D. avatar
⟳ GPU MODE retweeted
Tanishq Mathew Abraham, Ph.D.@iScienceLuvr·Aug 17

A very interesting blog post by @EliaIsPosting about applying autoresearch to a @GPU_MODE competition, winning 3rd place! Setup includes an orchestrator + subagents: idea generation, workers to implement the ideas, redteamers to audit the conclusions. Measuring actual signal and not regular variance is obviously a major challenge, which is highlighted in this blog post. The solution was an escalating evaluation protocol, which brings the comparison floor to ~0.15%. Link to blog post: elianaive.com/posts/another-…

Post attachment
sankalp avatar
⟳ GPU MODE retweeted
sankalp@dejavucoder·Aug 15

oh shit i am number 1 on hackernews atm (3x hackernews now lfg)

Post attachment

my latest blog post "auto-research with codex: how I achieved a 212x faster kernel over baseline with codex in GPU Mode's qr_v2 problem" is up now. in this post, i talk about my approach towards auto-kerneling on the QR decomposition problem. sankalp.bearblog.dev/autores…

Post attachment
Spectral Compute avatar
⟳ GPU MODE retweeted
Spectral Compute@SpectralCom·Aug 14

Today, our CTO @ChrisKitching17 joins @GPU_MODE - one of the sharper corners of the accelerated HPC community - for a live talk on YouTube. 📅 7-8:30PM London / 11AM-12:30PM SF Streamed via GPU MODE's Discord. If you spend your day in kernels, compilers, or wrestling GPUs from more than one vendor, come watch.

Post attachment
Stuart Sul avatar
⟳ GPU MODE retweeted
Stuart Sul@stuart_sul·Aug 4

(1/9) I'm thrilled to share the open-source release of Mixture-of-Kittens (MoK), our MoE megakernel for NVL72s! MoK fuses all mixture-of-experts communication and computation into a single, fully deterministic kernel, and powers Composer training across tens of thousands of GPUs. Joint work with @nash_c_brown, @hmwildermuth, @tmwilliamlin168, and @ellev3n11

We're open-sourcing Mixture-of-Kittens (MoK), our MoE training megakernel for NVL72s. It fuses all Mixture-of-Experts communication and computation into a single, fully deterministic kernel, and runs up to 2.37x faster than the strongest public baselines.

Post attachment
GPU MODE avatar
GPU MODE@GPU_MODE·Jul 29

Kinda wild that our community got together and collectively made Kimi inference on MI355X faster than B200 across all settings. Congrats to the winners team RadeonFlow and thanks again to AMD for working so closely with our community

Post attachmentPost attachment
Deepak Vijaykeerthy avatar
⟳ GPU MODE retweeted
Deepak Vijaykeerthy@deepakvijayke·Jul 24

If you'd like to know more, here's a nice talk at @GPU_MODE introducing the One Layer Deeper competition.🙂 youtube.com/watch?v=ustDJbxO…

Played around with this a bit. It’s nice to see folks thinking about it, and it is certainly an important problem to address. It's a deceptively good benchmark that asks whether gradient-based learning can learn a procedure for reusable computation that generalises to more steps than it saw during training, or whether it just learns a map between inputs and outputs. The question is whether the model can learn the serial computation itself, rather than relying primarily on test-time compute (in token space), and whether latent depth can provide a cheaper form of sequential computation. You can see the thought that has gone into designing the tasks. Kudos to folks @coreauto and @tilderesearch.

GPU MODE avatar
GPU MODE@GPU_MODE·Jul 20

We'll be doing a Q&A and demo tomorrow at 3:00pm Tuesday July 21 on our youtube channel. Please bring your questions!

Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable? @coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper" We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases. This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way. The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation. Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!

Post attachment
Bryce, the CUDA Colonel avatar
⟳ GPU MODE retweeted
Bryce, the CUDA Colonel@blelbach·Jul 19

Great results thus far from the 9 participants in the @GPU_MODE competition. 11B tokens and $500 in @modal credits thus far. Thanks to @damngamerz and everyone else for participating!

Post attachment

I'll be leading a sprint on the @GPU_MODE Cholesky competition at @europython / @EuroSciPy this Sat/Sun in Krakow! Come learn how to autoresearch and write CUDA kernels. @modal has generously donated GPU access and @nvidia will provide frontier LLM access for the sprint.

Mark Saroufim avatar
⟳ GPU MODE retweeted
Mark Saroufim@marksaroufim·Jul 18

Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable? @coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper" We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases. This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way. The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation. Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!

Post attachment
Mark Saroufim avatar
⟳ GPU MODE retweeted
Mark Saroufim@marksaroufim·Jun 26

Second problem is now out: dense symmetric eigenproblem A=QΛQT. Solution due on July 15! We've also enabled ncu profiling for your agents on a @verdacloud cloud box sponsored by our good friends at Brev at @NVIDIAAI

Launching a new kernel competition: Linear Algebra Kernels For The Age Of Research. First problem: batched QR decomposition on B200. Old math, modern hardware. Prize: Rare swag and hangout in SF

Tianqi Chen avatar
⟳ GPU MODE retweeted
Tianqi Chen@tqchenml·Jun 23

We taught a brand-new mini-series this year at @SCSatCMU on Modern GPU Programming for ML Systems, as part of the ML Systems course, touching on fun questions like what data layout swizzling is, how to use 3D TMA, and state-of-the-art Blackwell programming. We released a curated online book based on the materials: mlc.ai/modern-gpu-programmin… check it out

Post attachmentPost attachment
Bryce, the CUDA Colonel avatar
⟳ GPU MODE retweeted
Bryce, the CUDA Colonel@blelbach·Jun 21

This isn't one problem - it's a dozen or more! Specialized kernels for different shapes & kinds of matrices will win the day. Leveraging tensorcores & a mixture of numeric formats (FP64, FP32, TF32, FP16, FP8) is likely key to top answers. This may even end up memory bound.

Post attachment

We've released the QR problem, a more robust qr_v2 with a fresh leaderboard so please resubmit! Thank you to @blelbach, @myainotez and @nikhilbarhate99 for sharing feedback. Sorry if I missed anyone! I considered automatically backfilling all submissions but the rankings do change quite a bit so I figured a refresh would be better. Changelog * Fail submissions if they fail when we change random seeds * Add nasty correctness cases with more degenerate inputs in mixed batches * Recheck correctness when doing perf testing to avoid Volkswagen cheat * Reject Nan/Inf residuals * Validate each matrix factorization residual, since averaging was hiding bad matrices * Old QR is still open so folks can't see submissions but you can't submit anything to it Wontfix * Stream hacking is still banned via very blunt ban of the word "stream" we don't have a good solution for this * CUDA graphs are allowed but not particularly interesting to us Best submissions so far if I resubmit their solutions are

Post attachment
tender avatar
⟳ GPU MODE retweeted
tender@tenderizzation·Jun 16

one can dream

1:30