←
Vijay1,517 posts
Vijay avatar✓

Vijay ✓

@__tensorcore__

Systems and GPU Performance Mechanic - TBD Ex. CUTLASS 3.x / 4.x etc

◷ Joined July 2015▧ 60 Photos and videos
Vijay avatar
Vijay@__tensorcore__·Jan 23

As of last week, I am no longer at NVIDIA 🧵 Leaving the CUTLASS team was extremely hard. I will dearly miss my incredible colleagues and the extremely compelling mission statement of creating the world's best accelerator programming model w/ hardware software codesign 💚

Post attachment
Jensen Huang avatar
⟳ Vijay retweeted
Jensen Huang@JensenHuang·21h

Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen

Haicheng Wu avatar
⟳ Vijay retweeted
Haicheng Wu@asdf1234_0·21h

3 months ago, I joined Meta and teamed up with @__tensorcore__ again on MTIA. Strong products, strong roadmap. We are hiring seasoned engineers for all kinds of AI chip SW roles (compilers, kernels, runtime, performance model, etc.). Please send resume to acf@meta.com & DM me.

Lex Fridman avatar
⟳ Vijay retweeted
Lex Fridman@lexfridman·Nov 12

NVIDIA letting Jensen go is a huge mistake, for America and for humanity. It's Apple firing Steve Jobs all over again, but much bigger, given the role of AI in defining the future of our world. I've been keeping quiet on this out of respect for Jensen's family and team, who I've met and who are amazing humans, and because I personally hate infidelity. So my heart goes out to them in this difficult personal family moment. But I can also say that Jensen is human, and makes mistakes like all of us do. Not that I matter at all, but I've known Jensen for a long time and have gotten the chance to spend a lot of time with him, behind the scenes, away from mics and cameras. I have seen him show kindness, humility, care to everyone around him, including strangers, including from the poorest most struggling parts of society, treating all the same with the same thoughtfulness and attention, because they are fellow humans and not because he has something to gain from them. That shows rare character, to maintain humility and humanity through all of it. Another data point is what people who know him well have said privately to me about Jensen over the years. I've never had anyone tell me anything but how awesome he is or how much they love him. Probably over a thousand people at this point. This includes hundreds of engineers, CEOs, colleagues, friends, etc. Also, since there was some spy talk. Jensen is as proud an American as it gets. I'm in shock frankly at all of this, including the mass hysteria around it. I love America, and I think NVIDIA letting Jensen go is terrible for America. I stand with Jensen. Love you all ❤️

Alex Zhurkevich avatar
⟳ Vijay retweeted
Alex Zhurkevich@cudagdb·Nov 10

Of course that's your contention. You're a first year perf larpmeister. You just got finished reading some SGL and vLLM 100k LOC vibeslop megakernel PRs probably, you’re gonna be convinced of fwd pass megakernel next month when you get to programming guide, then you’re gonna be..

Post attachment
Alexandr Wang avatar
⟳ Vijay retweeted
Alexandr Wang@alexandr_wang·Nov 10

they told me size doesn't matter 😉 muse ember 🔥

0:25
Alexandr Wang avatar
⟳ Vijay retweeted
Alexandr Wang@alexandr_wang·Nov 9

1/ announcing our smallest, most efficient little model is here: muse ember 🔥 built for work that agents repeat all day, it gets you more intelligence per dollar than Haiku, Luna, and all other flash models at any reasoning setting and does it at a third of the cost or less!

Post attachment
Jon Barron avatar
⟳ Vijay retweeted
Jon Barron@jon_barron·Oct 23

Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:

0:07
driss guessous avatar
⟳ Vijay retweeted
driss guessous@drisspg·Oct 23

Gave a talk at PTC on flex_gemm which is a new gemm_epilogue API in pytorch built on Quack. I think its neat, not ground breaking but neat nonetheless :) github.com/drisspg/transform… planning to bootstrap this and continue to add fun examples

Alexandr Wang avatar
⟳ Vijay retweeted
Alexandr Wang@alexandr_wang·Oct 23

$100k to the developer who can make muse spark open fastest!

0:20
Sanket Purandare avatar
⟳ Vijay retweeted
Sanket Purandare@sanketpurandare·Oct 20

We’re sharing Dist-MoE, open-source library from Meta SuperIntelligence Labs (MSL @AIatMeta ). Dist-MoE provides fused communication and compute kernels for BF16/MXFP8 training and inference, plus NVFP4 inference. ✅ Fused communication and compute kernels ✅ Full CUDA graph support ✅ Dynamic activation packing across layers into a bounded buffer ✅ Dynamic, capacity-driven recompute ✅ Forward and backward kernels for BF16/MXFP8 ✅ No shared-expert branch required Some highlights from our GB300 NVL72 benchmarks: • Up to 1.40× speedup over Mixture-of-Kittens (MoK) in our imbalanced, ten-layer MXFP8 training workload across 64 GPUs. Dist-MoE packed activation state across all ten layers with zero capacity-induced recompute. • 3.09 useful PFLOP/s per GPU (3,090 TFLOP/s) for NVFP4 Mega inference across 16 GPUs. • 1.86 useful PFLOP/s per GPU (1,856 TFLOP/s) for MXFP8 Staged training across 64 GPUs. • 1.27 useful PFLOP/s per GPU (1,271 TFLOP/s) for BF16 training across 64 GPUs. • Sub-linear scaling under imbalance: in NVFP4 inference across 16 GPUs, 8× the received rows on the busiest GPU required 4.82× the iteration time; about 40% lower iteration time per received row. Full configurations, raw results, and measurement definitions are in the blog. @levendlee and I also put together an interactive visual explainer covering token dispatch and combine, quantization, forward/backward kernels, activation packing, dynamic recompute, and memory planning. Blog: meta-pytorch.org/dist_moe/pr… Repository: github.com/meta-pytorch/dist… TorchTitan (@PyTorch ) integration: github.com/pytorch/torchtita… If you work on MoE training, inference, or GPU kernels, take a look. We welcome feedback and contributions. #PyTorch #MixtureOfExperts #MoE #OpenSource #TorchTitan #DistMoE

Post attachment
Alexandr Wang avatar
⟳ Vijay retweeted
Alexandr Wang@alexandr_wang·Oct 16

after an exciting week of launches (muse wonder [full] & muse spark open), i want to write a note of appreciation to the amazing research teams at meta superintelligence labs on reaching such a special milestone ♥️ we started MSL and TBD a little more than a year ago and set a goal of producing a frontier model within a year or so. given where llama4 landed, that was an incredibly audacious goal, and we knew it wouldn't be easy. many people told us it would be impossible. we knew it was unlikely, but still worth doing. a remarkable group of researchers and engineers joined the effort from inside and outside meta. every single person came in, picked up a shovel, and got to work with earnestness and urgency. nobody asked whose job it was. we assembled a team of the most brilliant people i have ever had the opportunity to work with (and remember, i was an HRT intern 🙃). i am surrounded by ambitious, visionary, technically excellent, hard-working, pragmatic, thoughtful scientists, engineers, and product folks—people who will chase a weird loss spike past midnight just because they need to know why. these models are the product of their passion, hard work, and brilliance. somewhat paradoxically, when we started TBD, we set a principle of no deadlines. we knew we had an astronomical amount of work ahead of us—across pretraining, post-training, RL, multimodal, infrastructure, safety, data, AND frontier research. there was no use setting arbitrary timelines—we just had to get to work, be thoughtful, and build the foundations of a frontier research effort in record time. and run after run, this team did exactly that. our researchers are magicians. i can only describe building these models with this team as a wondrous journey of scientific discovery. i am constantly amazed by what we can accomplish with scaling. i will remember this past year for the rest of my life—the late nights, the broken runs, the evals that came back better than any of us dared to hope. watching this team create wonder after wonder has been a true and deep gift. more than just the models we released this week, we have built a machine that builds new models. it often sounds like a cop-out to say labs can just turn the crank to keep scaling, but now i have seen it first-hand. sure, it's a lot of work and a careful process, but we have built a scalable system to keep advancing at the frontier, and i am really proud of how fast this team keeps making these models better. lastly, i get to work with researchers who care deeply about how their work benefits humanity. it's easy to forget when everything is changing so fast, but the whole damn point is to make people's lives better. time and time again, i have seen our researchers, engineers, and product folks build our technology with care so it can truly be good for the world—whether that's building the best models for muse, thoughtfully open-sourcing strong models, or making sure the alignment and security of our systems are rock solid. this is a team of people who never forget there's a real person on the other side of every prompt. thank you to everyone at MSL for being part of this crazy journey over the past year. i'm so grateful for every one of you, and our work is not done!

Mark Zuckerberg avatar
⟳ Vijay retweeted
Mark Zuckerberg@finkd·Oct 15

Muse Spark Open -- weights releasing today -- runs great on new Mac Studios thanks to work with @johnternus and Apple.

Post attachment
Alexandr Wang avatar
⟳ Vijay retweeted
Alexandr Wang@alexandr_wang·Oct 15

1/ muse spark open is the most capable US open-weight model to date. 816B total params, 42B active. 1M-token context. natively multimodal. built for coding and agentic work like long-horizon tasks, hard software engineering, multi-step reasoning. america! 🇺🇲

Post attachment