Finally able to talk about what I've been heads-down on for 6 months at @nvidia 🦀⚡
We just open-sourced cuda-oxide — an experimental rustc backend that lets you write CUDA kernels in pure Rust.
No DSLs. No FFI. No source-to-source step. Single source.
Short🧵👇
This inspired me to try making a native CUDA/Metal version.
Highlights:
- 14x/24x faster than osam/pytorch on click
- 30-120fps masks on hover vs 1-10fps
- 66x faster segment everything (1.6s vs 105s)
- 2.9x/5.6x faster than pytorch/osam on apple silicon
cargo add sam3rs / pip install sam3rs
github.com/avencera/sam3rs
I replaced ONNX Runtime, cuDNN and cuBLAS in speakrs with GPU kernels written in Rust. Same DER, about 60× faster than pyannote on the same GPU. Read more full results, what went well and lessons learned,
I replaced ONNX Runtime, cuDNN and cuBLAS in speakrs with GPU kernels written in Rust. Same DER, about 60× faster than pyannote on the same GPU. Read more full results, what went well and lessons learned,
github.com/bitemyapp/inferro…
Research-grade only, but ~182 tokens per second with qwen3 8b w8a8 on an RTX 6000 Pro with all-Rust GPU kernels is nothing to sniff at either.
More to come later on this and some orthogonal things.
Still on a terminal kick - built a Google Maps renderer in my CLI!
From scratch - single megakernel in Rust/cuda-oxide for rasterization, renders through Kitty graphics.
My favorite talk from Scala Days 2026 was about baking refinement types right into the compiler.
Unlike libraries like Iron or Refined, it adds a built-in solver that verifies logic and math across multiple variables at compile time with zero runtime cost.
Moving from simple checks to full verification is how Scala's type system should evolve to constrain LLMs - AI code won't compile unless the math holds up.
The latest Burn release includes a migration of CubeCL's compiler infrastructure to use pliron!
github.com/tracel-ai/cubecl/…
It's nice to see pliron being adopted by Rust compiler projects, with both CubeCL and cuda-oxide now building on it.
Burn 0.22 is out: our biggest release yet, and a big step towards 1.0.
Models no longer carry a backend generic. That alone makes rebuilding after a model edit 6× to 15× faster.
Start learning GPU programming with one fully animated YouTube lecture.
youtu.be/dEZP1qUNTOk
This is the first video of my GPU programming series with Triton.
Every idea in it is visual and animated, just like the GIF in this post, but for a full hour.
You write c = a + b in PyTorch every day.
This lecture shows what the GPU actually does with it.
𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐥𝐞𝐜𝐭𝐮𝐫𝐞 𝐜𝐨𝐯𝐞𝐫𝐬
- How a GPU sits inside your computer: PCIe, NVLink, servers and clusters
- Inside an NVIDIA H100: SMs, FP32 lanes, the warp scheduler, tensor cores, shared memory
- Threads, warps, blocks and grids, and how each thread finds its element
- Memory: SRAM vs DRAM, DDR vs GDDR vs HBM, caches, latency vs bandwidth
- Making memory fast: latency hiding, coalescing and kernel fusion
- Why Triton exists, and how it lets you think in tiles instead of threads
- Memory-bound vs compute-bound, arithmetic intensity and the roofline
- Attention, the N x N matrix, and how FlashAttention fixes it
No CUDA or hardware background needed.
I want to make more long-form technical animated videos, something similar to what @neural_avb does.
Taken inspiration from him
𝐖𝐡𝐚𝐭'𝐬 𝐧𝐞𝐱𝐭
- Lecture 2: writing our first Triton kernel, vector add, line by line, will post it in few days
- More lectures on GPU programming and inference optimization
- A planned series on deep learning for audio
Subscribe to the channel so you don't miss them.
Please also share the video with your ML and DL friends, or in any study or community groups you're part of.
Watch the full lecture:
youtu.be/dEZP1qUNTOk
If you are into audio deep learning, inference, or Triton, follow @Mayank_022 me.
I will be uploading more posts related to that.
We’ll cover:
• 🗺️ Finding your way around the repo
• 🔍 What happens behind #[kernel]
• 🚀 How the host launches your GPU kernel
youtu.be/3X_x0MfN90Q?si=fRZZ…
🦀 A question I keep getting: “How do I get started with cuda-oxide?”
With, Rust, GPUs, and compilers, it can be a lot to take in.
Here’s my attempt to make it easier: a simple Rust GPU kernel, what #[kernel] generates, and how the host launches it.
The allocator_api feature has been stabilized!
> Rust’s Allocator API standardizes custom memory allocation, enabling collections to use specialized strategies for better control, performance, and memory efficiency.