Last Year We Built Megakernels. This Year We Deleted Them.
One year ago, we helped introduce some of the first AI megakernels to the world. Since then, we’ve enjoyed seeing their wide adoption from friends and researchers. However, while whole model megakernels are a helpful abstraction, we’ve found that they overcomplicate and are not always an ideal optimization target.
Below, we discuss our journey to understand and push past these limitations. We start with a deep dive into two contrasting paradigms of ML optimization – Torch Compile and Megakernels
Together Compiler (TKCC) generalizes beyond LLMs to ML models in PyTorch to significantly outperform the PyTorch fwd pass baseline but also a handwritten claude code harness to optimize the baseline.
together.ai/blog/together-co…
And because TKCC makes stock SciML models so ⚡️ fast, something I fully suspect is possible is that we can train a model to use higher resolution inputs or smaller patches for the same number of GPU hours.
3/n
TKCC also unlocks performance gains. For example, on weather models, we can use our 3x speedups to process 3x more inputs, and thus have less statistical noise in our final mean temperature predictions.
Our TKCC runs here use less wall clock time than the stock models! 2/n
Congratulations to @reyna_abhyankar and @dylan__lim for their >1yr effort on Together Compiler!
Their work was so incredible that I found it generalized well beyond LLMs to SciML models 🔬.
Here it is outperforming Claude on nearly every model I tried
1/n
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar@dylan__lim@realDanFu@togethercompute!!
This project was a blast to work on! Here's another cool 1 Trillion X zoom into a fractal from the Equilibrium Reasoners model for Sudoku (plus some more visuals in the thread)
I am very grateful to @unconvAI for supporting my group's work on fractal basins in reasoning models! this proposal is what first inspired us to study reasoning as a dynamical system. looking forward to learning a lot more from the company & the other awardees this year
Our group discovered that reasoning models produce fractals when asked to solve hard problems. We can use nonlinear dynamics to probe the thinking processes of recurrent depth models on Sudoku, mathematics, and even ARC-AGI (1/N)
arxiv.org/abs/2609.04963
AI compute and inference are increasingly $$$. How can we change the unit economics of AI to improve accessibility? It's been fun working with @prlnet to release the first model endpoint that simultaneously generates tokens **and** a digital asset that can subsidize inference! 🪙 Check it out, links below 🚀
Loved working on this project, super proud of it! Thank you @AwesomeBao, @bialjail, and @wgilpin0 for being such a great team that I learned so much from.
Excited to share our work at ICML!
We trained LoRA adapters of different ranks to understand training dynamics, finding that adapters for GSM8k live in a surprisingly vast, low-rank solution space.
This hints that some model skills are easy to learn, and training is more forgiving than we think. @hasith_v 1/6 🧵
We post-trained MedGemma to be SoTA in visual medicine ddx, outperforming Opus 4.6, Gemini 3.1 and GPT-5.4 while running at ~1/30th the cost. @nollahealth Part 1 - improving visual reasoning 🧵1/6