←
Rochelle Mattern50 posts
Rochelle Mattern avatar

Rochelle Mattern

@ro_mattern

San Francisco// AI Solutions Engineering

◷ Joined December 2009▧ 1 Photos and videos
Simran Arora avatar
⟳ Rochelle Mattern retweeted
Simran Arora@simran_s_arora·Nov 4

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

Reyna Abhyankar avatar
⟳ Rochelle Mattern retweeted
Reyna Abhyankar@reyna_abhyankar·Nov 4

I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference. Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it! Paper: alphaxiv.org/abs/2611.togeth… It was super exciting to build this from scratch :) more thoughts in thread below!! 1/

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

Dylan Lim avatar
⟳ Rochelle Mattern retweeted
Dylan Lim@dylan__lim·Nov 4

A little over a year ago, I started my journey into kernels and inference optimization. Since then, the field has rapidly evolved, and many tasks like kernel writing have become increasingly obsolete. I’ve spent a lot of time reflecting on ML Systems in the agentic era, and how such systems must be designed to efficiently and thoroughly explore the optimization space. I’m proud to announce Together Compiler, a continuous optimization system for the age of polymorphic inference!

Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!

Deon Nicholas avatar
⟳ Rochelle Mattern retweeted
Deon Nicholas@dojideon·Oct 20

Espa is now generally available! Personal AI agents are here, and I sat down with @BloombergTV @RomaineBostick to discuss what will decide which ones people actually let into their lives. Our goal is to be the most trusted AI assistant for life and work.

0:47
Together AI avatar
⟳ Rochelle Mattern retweeted
Together AI@togethercompute·Sep 23

We're releasing tev1-4B-experimental, a Jev-like classifier finetuned on top of Qwen3.5 4B. We're making it available on Together serverless at $0.042/M input & $0/M output. Also releasing the data recipe & a tutorial on how to finetune your own (Tev1 cost $17 to train!).

Hassan @nutlope Sep 23 Article How to train your own Jev for $17 TLDR: We just launched our own Jev-like classifier, together/Tev1-4B-experimental, on top of Qwen3.5 4B. In this blog post we’ll show you how to fine-tune your own version! Jev has quickly become one

Together AI avatar
⟳ Rochelle Mattern retweeted
Together AI@togethercompute·Sep 22

New on Dedicated Model Inference: canary rollouts. Upgrade the model behind a live endpoint without downtime. Traffic moves from your current deployment to the new checkpoint in gated steps (default 5% → 25% → 50% → 100%). Health checks run before any traffic shifts. After every step, metric gates compare the new model's p95 latency and error rate against the old one. If a gate trips, the rollout pauses at the canary share and waits for you: resume, promote to 100%, or roll back. Three strategies: canary, blue-green, and rolling. Available now via the tg CLI, REST API, and Python SDK. Learn how to start a rollout: together.ai/blog/canary-roll…

Post attachment
Kimi.ai avatar
⟳ Rochelle Mattern retweeted
Kimi.ai@Kimi_Moonshot·Jul 27

Kimi K3 is now live on @togethercompute! Happy to have Together AI as our day0 launch partner, giving developers immediate access to K3 through high-throughput inference optimized for coding agents and production workloads. A huge thank-you to the Together AI team for moving fast and making this launch possible!

Post attachment

Kimi K3 is now live on Together AI. We’re proud to be a Day 0 launch partner for @Kimi_Moonshot’s open frontier model, built for long-running agentic workflows across code, tools, vision, and research.

Post attachment
Abhishek Das avatar
⟳ Rochelle Mattern retweeted
Abhishek Das@abhshkdz·Apr 23

We're excited to launch Delegate. An agent you delegate work to and move on with your life.

1:17
Together AI avatar
⟳ Rochelle Mattern retweeted
Together AI@togethercompute·1 Dec 2025

Introducing Trinity Mini from @arcee_ai, an open-weight 26B sparse MoE model that activates just 3B parameters per token while delivering frontier-class reasoning. AI natives can now use Trinity Mini on Together AI — and benefit from reliable inference for production-scale agentic workflows.

Post attachment
Drishan Arora avatar
⟳ Rochelle Mattern retweeted
Drishan Arora@drishanarora·19 Nov 2025

Today, we are releasing the best open-weight LLM by a US company: Cogito v2.1 671B. On most industry benchmarks and our internal evals, the model performs competitively with frontier closed and open models, while being ahead of any US open model (such as the best versions of OpenAI’s GPT-OSS, Nvidia’s Nemotron and Meta’s Llama). We also built an interface where you can try the model (it’s free and we don’t store any chats): chat.deepcogito.com Additionally, you can download the model on @huggingface, or try it out on @openrouter, @togethercompute, @FireworksAI_HQ , @ollama cloud, @runpod, @baseten, or run it locally using @ollama or @UnslothAI. This model uses significantly fewer tokens amongst any similar capability models, because it has better reasoning capabilities. You will also notice improvements across instruction following, coding, longer queries, multi-turn and creativity. 📌 Model Weights: huggingface.co/collections/d… 📌Openrouter: openrouter.ai/deepcogito/cog… 📌 HF Blog: huggingface.co/blog/deepcogi… Some notes on our approach + design choices below 👇

Post attachmentPost attachmentPost attachment
Cursor avatar
⟳ Rochelle Mattern retweeted
Cursor@cursor_ai·29 Oct 2025

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

1:04
Sophia Xing avatar
⟳ Rochelle Mattern retweeted
Sophia Xing@xingsthatmatter·10 Oct 2025

I’ve always loved stories, the kind that make you feel something, that linger long after the credits fade. Plot Party is an agentic canvas built for visual storytelling, the easiest way to turn imagination into cinema. We are now in private beta. @JoinPlotParty

0:47
BLACKBOX AI avatar
⟳ Rochelle Mattern retweeted
BLACKBOX AI@blackboxai·10 Sep 2024

The team at @SambaNovaAI are GETTING AFTER IT! Some people believe in weekends, Blackbox AI and Sambanova don't! We met over the past 2 weekends to test at scale and finalize our partnership. Results are LEGIT! more to come…

SambaNova@SambaNovaAI10 Sep 2024

📣 Announcing the world’s fastest AI platform SambaNova Cloud runs Llama 3.1 405B @ 132t/s at full precision ✅ Llama 3.1 405B @ 132 tokens/sec ✅ Llama 3.1 70B @ 570 tokens/sec ✅ 10X Faster Inference than GPUs Start developing #FastAI ➡️ cloud.sambanova.ai @AIatMeta

Post attachment
0:08