data center builders must design for the future of power-intensive racks if they want to capture AI demand
nvidia’s next rack (kyber) needs 600 kw, and data centers today are not built for this. the average rack in existing data centers gets about 8 kw. very difficult to increase power in a data center that's already built
the pattern is that each chip generation needs more power per rack.
- for nvidia, hopper was ~40 kw, blackwell at ~120 kw, and kyber will be at 600kw
the power shortage is more nuanced than the quantity of power. can power reach the racks that need it?
the biggest customers, like the labs, buy the newest chips first. to capture their demand, data centers should be designed differently
Excited to announce our collaboration with @Supermicro, the perfect partner to build and support inference racks powered by SambaNova SN50.
Our racks join Supermicro's Data Center Building Block Solutions portfolio. They fit the air-cooled data centers customers already run, next to the GPUs they already own. This first batch of SN50s is only phase one. Pumped for what comes next!
sambanova.ai/press/sambanova…
Thank you @ClayMagouyrk for the shoutout at the Oracle AI World stage this morning!
We’re proud to bring fast, high-performance AI inference to @Oracle Cloud Infrastructure (OCI).
More on our partnership below ⬇️
LIVE from Oracle AI World!
SambaNova and @Oracle are collaborating to bring high-performance inference to Oracle Cloud Infrastructure (OCI).
“Our customers want the right infrastructure for every AI workload, and inference demand is growing fast. We plan to bring SambaNova’s RDU chips to OCI to give AI builders a high-performance, efficient way to run models at scale.” — @MThiagarajan_, President, OCI.
Full press release here: bit.ly/4e0V3Ts
🚨 Big news today: We’ve raised $100M for our Series A! 🎉 🚀
We believe demand for fast inference is many times larger than demand for slow inference, and it’s about to go exponential.
Thanks to our co-leads @BlackbirdVC and @Abstractvc. Two visionary partners join our journey 🎉 🚀
Disaggregated inference is the future.
Great panel with @general_compute discussing their heterogenous compute cloud, the speciation of compute for disaggregated inference, and how thermo compute will fit in the picture
More coming soon. Stay tuned 😉
A look under the hood of SN50.
@raghup17 took the stage at @hotchipsorg for a deep dive into the architecture behind our latest RDU and what it takes to scale dataflow for production inference.
Thanks to @RyanSmithAT for the great technical breakdown.
Read it here 👇
bit.ly/4gT6C0O
A look under the hood of SN50.
@raghup17 took the stage at @hotchipsorg for a deep dive into the architecture behind our latest RDU and what it takes to scale dataflow for production inference.
Thanks to @RyanSmithAT for the great technical breakdown.
Read it here 👇
bit.ly/4gT6C0O
Many thoughts for @SambaNovaAI perspective...
Firstly, this is great validation that kernel-by-kernel execution models will never suffice for fast inference. CUDA is a kernel-by-kernel paradigm, so for fast inference, there is no CUDA moat.
Secondly, TileRT is dataflow emulation and use some of the same concepts we do in our compiler. However, doing this on hardware that has been built for kernel-by-kernel execution is inherently limited and this shows up in a few different ways. As described below:
1. Can they increase model parallelization to get faster? There is very little evidence of any GPU being able to scale tensor parallel beyond 8 GPUs, despite the fast that there is a 72-GPU scale-up domain readily available now. On InferenceX, no-high speed decoding use case uses more than 8 GPUs (data parallel doesn't count). In fact there is still a lot of HBM this is left unutilized in the TP8 mappings that TileRT has shown. Unless they can show an ability to saturate HBM bandwidth within a large model parallel domain, they will not keep up with true dataflow chips.
2. Can they batch without killing interactivity AND at long context length (truly Premium Inference)? TBD but my feeling is that this will be extremely difficult using dataflow emulation on GPUs, even on Rubins. On SN50, we will be able to reach the same speed with much higher batches per chip and at the 140K median context length that has become the default on InferenceX.
3. Can they generalize? The article rightly points out how hard it is to generalize - "a persistent engine kernel means the model is statically expanded ahead of time into one resident program". This is indeed extremely hard. We have been working on this problem for 9 years in our compiler. We haven't solved all these problems yet but are extremely well placed to be the first that does. We also have the right hardware target that makes the problem much easier than trying to retrofit GPUs to do this.
The most salient point in the article is the fungibility that this enables, but ultimately I think this will give buyers more reason to buy dataflow chips, as long as they do not over-specialize. i.e. if I am a AI inference provider and I buy some systems for prefill and high throughput decoding, and some systems to fast decoding, and my workloads change to require more fast decoding, I can reallocate my GPUs to service this demand with TileRT, albeit at a much higher cost per 1M tokens. Similarly, if the workloads move in the other direction and I need more high/medium throughput and low/medium interactivity, I should be able to reallocate my dataflow chips for this and offer good enough tokenomics. This is exactly how we think of the problem, as you need fungibility at the edges, not at the extremes, and you want to cover enough of the pareto curve to be able to adapt, but still enough specialization to differentiate.
This is an important milestone for SambaNova. Not only is it the first verified pareto curve SemiAnalysis has shared (albeit unofficial) of any AI chip startup, but it is our first demonstration of SN50 system performance, as well as a real working demonstration of heterogeneous inference with Nvidia GPUs.
But there is a lot of work for us to do. We will scale up our mappings with expert and tensor parallelism to increase speed and per chip throughput, and continue to push the pareto frontier.
Great work to Sambanova team on bringing up SN50 MVP demo of MiniMax M2.7 model Excited for when the software stack will work above batch size 2 & will work on more frontier models!! The sambanova software team is def grinding!
SHOCKING: Many neocloud executives we spoke with feel that if they have non-NVIDIA networking gear on their cluster, or if their cloud has an AMD GPU or TPU offering, NVIDIA will retaliate. They feel that retaliation includes not giving early allocation or no longer supporting a potential IPO/VC raise.(1/3)🧵
Intel and @SambaNovaAI are partnering to deliver rackscale AI infrastructure for inference and agentic workloads, built on Intel Xeon processors and SambaNova RDUs
Watch the Computex keynote: ms.spr.ly/6013vbmYS
Never has it been clearer that SambaNova has the best architecture in the world for AI inference. This will be proven to be true over the next 6 months.