←
Stu Cao1,176 posts
Stu Cao avatar✓

Stu Cao ✓

@caoshuxun

Inference @primeintellect

◷ Joined August 2010▧ 128 Photos and videos
Stu Cao avatar
Stu Cao@caoshuxun·Oct 4

Two months ago, I left Annapurna Labs, ending a five-year chapter at Amazon. Amazon treated me exceptionally well and gave me more opportunities to grow than I can count. I’m grateful for the people I worked with and everything I learned there. I followed my heart to Prime Intellect. Its mission to build an open superintelligence stack deeply resonated with me. I believe every person and every business should be empowered to build their own intelligence. Your AI agent should be yours, not a third-party stranger. Walking into Prime’s SF office, I felt ten years younger. These past two months have been the most exciting of my career. Three things stand out: 1. The people. Prime’s team doesn’t all come from famous AI labs, but I’ve rarely seen this level of curiosity and determination. When we made our high interactivity GLM-5.3 serving endpoint available internally, everyone switched their agents to it. They never went back. The speed improved productivity. Just as importantly, the team kept sharing feedback, helping us refine performance and tool-calling reliability. Soon, RL rollouts, synthetic data generation, and long-horizon agent research were running on the same infrastructure. As of this week, our internal inference usage is 600–700 billion tokens per day. That experience gave us the confidence to open the service to the public. 2. The full stack. Prime brings together hosted training with prime-rl, verifiers, sandboxes, an RL environment hub, the Prime Agent harness, and now Prime Inference. You can train, deploy, and run agents in one place, without stitching together infrastructure through a massive engineering effort. And most of these components are fully open source. Our inference stack builds on that same foundation, with deep collaboration across NVIDIA Dynamo, Inferact, and the broader vLLM community. 3. The compute. For anyone working on LLM infrastructure, access to compute at this scale is a privilege. Building and operating inference across large scale of GB200 racks and B300s is not something I take for granted. Vera is already in our hands, and Rubin is on the horizon. NVIDIA’s latest hardware gives us the foundation to push inference performance further. Alongside the public launch, we published a technical post on how we built Prime Inference: primeintellect.ai/blog/prime… Inference has moved incredibly fast over the past year, and it can be hard to separate useful advances from the noise. We use GLM-5.3 optimization to show the work end to end: how we serve agent workloads with low latency while also supporting high concurrency, dependable availability, and reliable tool calls. Recommend for a read if you are interested in learning about P/D, topology choice, NVFP4 KV compression kernel fusion, tool call reliability. Fast inference matters. Serving real agents reliably at scale is what counts.

Garry Tan avatar
⟳ Stu Cao retweeted
Garry Tan@garrytan·8h

Only three people to remove from the board? Easy work 15% shareholder vote coming

Jensen Huang avatar
⟳ Stu Cao retweeted
Jensen Huang@JensenHuang·20h

Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen

vik avatar
⟳ Stu Cao retweeted
vik@vikhyatk·Nov 17

It's a good model. (And supported on Prime Inference.)

Post attachmentPost attachment
Stu Cao avatar
Stu Cao@caoshuxun·Nov 15

There are certainly a lot of good work here but we just have too much benchmaxxing in inference right now. training a speculator on the benchmark dataset just to game the numbers. I also noticed many benchmarks published today use P90, which doesn’t really represent the speed every user can experience.

Tim Sweeney avatar
⟳ Stu Cao retweeted
Tim Sweeney@TimSweeneyEpic·Nov 13

You sort of set him up, didn’t you, by creating the impression that senior government officials anonymously provided information about spying, when in fact the info came from a lobbyist at a competing AI company?

CEOInterviews.AI avatar
⟳ Stu Cao retweeted
CEOInterviews.AI@CEOinterview·Nov 13

Dylan Patel says Jensen personally talked tiny suppliers into letting Nvidia buy up their supply, and with him gone Google is already begging them for capacity "Jensen had complete foresight and understood the demand required and went to all of these executives, even of tiny companies where they only generate a few hundred million dollars of revenue but they have an important subcomponent." "He convinced them to let him buy the supply and build it all out..." "And now with Jensen gone, these companies are looking around the market and it's like, well, Google's begging for more capacity from the whole supply chain." "And without Jensen, can they maintain technology advantages given they're only a year ahead of their biggest competitors? It's not like the gap is massive."

1:31
1:31
Elon Musk avatar
⟳ Stu Cao retweeted
Elon Musk@elonmusk·Nov 13

A petition is underway to reinstate Jensen and I’m signing it too

Daebak 앤디 avatar
⟳ Stu Cao retweeted
Daebak 앤디@MyDaebakCafe·Nov 13

I am actually disgusted by the timing of this drop. And this will get a bit more technical. Let’s do a real technical teardown of this "1,385-sigma Blackwell weight fingerprint" report that OpenAI and Anthropic are breathlessly promoting today, because banking on nobody reading past the headline is how DC narratives get manufactured. ​First, look at the actual CUDA maths: when PyTorch initialises model weights (torch.nn.init) with a fixed PRNG seed, curand/Philox parallelises random number generation across the GPU’s Streaming Multiprocessors (SMs). Because Hopper has 132 SMs and Blackwell has 148 or 152 SMs, the exact same seed lays out initial weights in a different spatial pattern depending on the GPU’s SM count. In huge MoE models, certain low-gradient null-spaces or unrouted slices preserve faint directional traces of those step-0 weights even after training. ​So yes, brute-forcing the seed and SM grid can show that a model’s step_0 weight initialisation script ran on a 148/152-SM Blackwell GPU. Now read what the summary explicitly admits in the fine print: 1. Initialisation is not full training: The report literally states that these signatures show how the weights were created, and "support (but do not prove) the inference that Blackwell also participated in model training". Initialising a model checkpoint takes seconds on a single rented dev node before loading the safetensors onto a standard Hopper cluster. The "2,000 to 3,000 Blackwell GPUs" and "$500 million" cluster numbers do not come from the weight fingerprint and Depue admits in his own thread that he simply "estimated" cluster sizes from release schedules and model details. ​2. Zero geolocation and zero proof of smuggling: The final sentence of the executive summary explicitly admits that the findings "do not establish the physical location of the chips, how they were acquired, or whether export control regulations were violated". Depue himself rented a cloud GB200 instance just to run his verification test. Commercial labs that are not on the Entity List renting bare-metal Blackwell nodes in offshore clouds (Singapore, Japan, or the Middle East) is completely legal under BIS rules and leaves the exact same SM fingerprint. Now ask yourself why a former OpenAI researcher dropped this on November 12, the exact day Jensen Huang was pushed out, with OpenAI and Anthropic immediately stepping in to "independently verify" it. US closed-source labs are watching their paid API moats get eaten alive by free Chinese open-weights models. Conflating a step-0 PRNG initialisation seed with a "$500M smuggled Blackwell cluster" hands DC hawks the exact pretext OpenAI and Anthropic want to regulate global cloud compute and kneecap open-source AI.

Post attachment
Stu Cao avatar
Stu Cao@caoshuxun·Nov 13

If you ever look into DeepInfer source code, it cannot even run on Hopper. Blackwell and Ascend only.

Post attachment
LightSeek Foundation avatar
⟳ Stu Cao retweeted
LightSeek Foundation@lightseekorg·Nov 13

Before May, beating TensorRT LLM seemed hard. We open-sourced TokenSpeed in May to show it isn’t: systematic profiling and tuning is enough. Now coding agents write fast kernels. Top performance is just tokens + GPUs + time. Conviction is everything.

何川(复利版) avatar
⟳ Stu Cao retweeted
何川(复利版)@daxiang555·Nov 13

黄仁勋和女儿黄敏珊在内地节目《波士堂》唯一一次同台亮相。 黄仁勋居然是乒乓球高手,并且热爱做饭

3:20
Stu Cao avatar
Stu Cao@caoshuxun·Nov 12

As both a user and an inference provider, I have to say @OpenRouter is one of the most customer-obsessed companies I’ve ever worked with. As a user, OpenRouter team consistently provides valuable insights on optimizing token costs through smarter provider selection and routing. As an inference provider, the growth has been staggering. Since onboarding in late September, we scaled from 0 to 100B daily tokens in our first month, then from 100B to 600B daily tokens in less than two weeks. A true rocket ship of a platform. 🚀

Post attachment
0:45
Prime Intellect avatar
⟳ Stu Cao retweeted
Prime Intellect@PrimeIntellect·Nov 9

Prime Inference has the highest cache rate of any third-party provider for DeepSeek V4.1 Flash 1017 High cache hit rate means that more of your prompt is billed at the cache-read price This offers the lowest effective input cost on the board: $0.00443/M

Post attachment
elie avatar
⟳ Stu Cao retweeted
elie@eliebakouch·Nov 6

inkling 2 small is extrememly impressive it's a new pre train from scratch (17B active 594B total param), train fully in only a few dozen of days, with much much much less compute and data (only 12T token!) than any other model they are competing with. the base model ppl loss on their internal code is lower than Kimi K3.1 (they even "base-recovered" the model to make a fair comparison!!) the benchmark are super strong overall and they also optimize for metric and domain that they care, like finetunability and domain like finance huge congrats, imo one of the most impressive release in the open weight western ecosystem this year thinkingmachines.ai/news/ink…

Post attachment