←
Dan Fu971 posts
Dan Fu avatar✓

Dan Fu ✓

@realDanFu

Founding SVP of Research @togethercompute Assistant Professor @ucsd_cse Looking for talented kernel engineers and performance engineers!

◷ Joined September 2019▧ 196 Photos and videos
Dan Fu avatar
Dan Fu@realDanFu·19 Aug 2024

Excited to share that I will be joining UCSD CSE as an assistant professor in January 2026! I'll be recruiting PhD students from the 2024 application pool - if you're interested in anything ML Sys/efficiency/etc please reach out & put my name on your application! Until then I'll be finishing up some requirements at Stanford (long story...) and hanging out at @togethercompute. Stay tuned for more!

Dylan Lim avatar
⟳ Dan Fu retweeted
Dylan Lim@dylan__lim·Nov 16

Last Year We Built Megakernels. This Year We Deleted Them. One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target. Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels

Post attachment
1:28
Together AI avatar
⟳ Dan Fu retweeted
Together AI@togethercompute·Nov 11

tl;dr of what Together Compiler does, from @realDanFu on @thecompute100: 1/ run the model end to end 2/ find the bottlenecks 3/ let the model fix them Links below 👇

0:33
Dan Fu avatar
Dan Fu@realDanFu·Nov 10

Fun podcast on kernels, origins of @togethercompute, and how we’re using AI to build inference in an RSI inference loop with Together Compiler. Thanks so much for having me @gabyllorenzi and @BSchech!

1:04:58
The Compute 100 avatar
⟳ Dan Fu retweeted
The Compute 100@thecompute100·Nov 10

Today’s guest is @realDanFu, SVP of Research at @togethercompute. A co-creator of FlashAttention, Dan now leads research at a company that provides the compute and software to train and serve AI models. From GPU instructions to the economics of open models, we discuss: - What CUDA, kernels and compilers each do - Why FlashAttention rethought data movement - How AI is changing kernel engineering - How new open models get to market - How inference revenue funds open models - Owning versus leasing GPUs Timestamps: 0:00 Introduction 1:04 From AI models to CUDA kernels 11:38 Launching new models at Together 22:36 How model architectures are changing 27:42 Comparing open and closed models 31:57 How AI is changing kernel engineering 38:01 From systems research to FlashAttention 47:11 Using AI to improve inference 48:16 Together’s origins and open models 51:17 Owning versus leasing compute 52:55 Building the research team and new chips 54:56 Frontier labs, open models and partnerships 57:16 Inference profits and funding the next model 1:00:13 Academic research and new talent 1:02:47 Robotics and AI in the physical world Listen to the show, and follow along for new episodes every Tuesday.

1:04:58
Tri Dao avatar
⟳ Dan Fu retweeted
Tri Dao@tri_dao·Nov 5

Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share

Together AI avatar
⟳ Dan Fu retweeted
Together AI@togethercompute·Nov 4

Inference is entering a new era. Models ship weekly, traffic shapes shift daily, and hardware turns over every year. The era of Static Inference, tuned once for one model with one traffic shape on one chip, is ending. Welcome to the era of Polymorphic Inference: inference that adapts to new models, workloads, and hardware as they arrive. Today we’re announcing Together Compiler, our first step into that era. It enables continual optimization, so a single overnight run can match or beat the manual expert tuning that used to take weeks. Read the full vision from our SVPs of Research, @simran_s_arora and @realDanFu, on Polymorphic Inference and our research roadmap 👇 togetherai.link/uQJRZjW

0:14
Dylan Lim avatar
⟳ Dan Fu retweeted
Dylan Lim@dylan__lim·Nov 4

A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!

Reyna Abhyankar avatar
⟳ Dan Fu retweeted
Reyna Abhyankar@reyna_abhyankar·Nov 4

I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference. Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it! Paper: alphaxiv.org/abs/2611.togeth… It was super exciting to build this from scratch :) more thoughts in thread below!! 1/

Dan Fu avatar
Dan Fu@realDanFu·Nov 4

A little over a year later, it’s super gratifying to see those efforts come to fruition in production. We learned a ton of lessons building the world’s first production-scale inference RSI system - sections 3 and 4.4 are especially interesting for those production insights. 4/

Post attachmentPost attachment
Dan Fu avatar
Dan Fu@realDanFu·Nov 4

We had the first inklings last summer that it might be time to let go of all these abstractions with the new strength of AI agents. We tasked @reyna_abhyankar (then a PhD student and intern) with - “what if we just do all these kernels automatically?” 3/

Dan Fu avatar
Dan Fu@realDanFu·Nov 4

This is the culmination of a few years of our research agenda on how to scale inference optimization. It builds on a lot of our earlier work on abstractions and kernel programming fundamentals (from the Flash series of kernels to ThunderKittens, megakernels, and more). 2/

Dan Fu avatar
Dan Fu@realDanFu·Nov 4

Super excited to (officially) share Together Compiler with the world! This is our opinionated take at @togethercompute about how to scale inference in an increasingly polymorphic world (many models, archs, traffic, hardware). Deployed across our inference fleet today 🚀 1/

Simran Arora avatar
⟳ Dan Fu retweeted
Simran Arora@simran_s_arora·Nov 4

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!