←
Desh Raj2,487 posts
Desh Raj avatar

Desh Raj

@rdesh26

Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati

◷ Joined September 2009▧ 204 Photos and videos
Desh Raj avatar
Desh Raj@rdesh26·Jan 12

I’m happy to share that I’m starting a new position as Senior Research Scientist at @nvidia! Looking forward to open science for speech full-duplex models :)

Post attachment

After 2 wonderful years, I left Meta this week. During this time, I worked on several projects related to speech and LLMs: - Built the first multi-channel audio foundation model with M-BEST-RQ (arxiv.org/abs/2409.11494) - Made ASR with SpeechLLMs faster (arxiv.org/abs/2409.08148) and more accurate (ieeexplore.ieee.org/document…) - Shipped the first production-ready full-duplex voice assistant (about.fb.com/news/2025/04/in…) - Improved Moshi’s reasoning capability with chain-of-thought (arxiv.org/abs/2510.07497) I am grateful to my managers for having my back on critical projects, and fortunate to have collaborated with several brilliant researchers and engineers during this time. As to what's next, I am still in NYC and continuing to do speech research. More on that later!

Post attachment
Desh Raj avatar
Desh Raj@rdesh26·2h

Finished ICASSP reviews: 2 weak accept, 1 weak reject, 5 strong reject. It definitely felt like the worst batch of papers I have reviewed at ICASSP, so I computed some stats from my older reviews. And indeed, it seems the % of "definite rejects" is much larger this time around.

Post attachment
Gradium avatar
⟳ Desh Raj retweeted
Gradium@GradiumAI·11h

How do you describe a voice? We learned a lot by designing and testing 16,000+ voices while building our catalog, and from the prompts and feedback that teams building with Gradium share with us. The gap between the voice in your head and the one you get usually comes down to a few words. Our best prompts cover five parts: who is speaking, their role, accent, mood, and how they sound. We put it all in one post, with samples to compare: gradium.ai/blog/how-to-descr… What's the most unusual voice you've tried to design?

1:17
Desh Raj avatar
Desh Raj@rdesh26·Nov 17

Voice AI has a vocabulary problem, so I made a 12-min primer to set records straight! Disclosures: My ideas/direction + Fable's implementation + Gradium's voiceover.

12:20
Desh Raj avatar
Desh Raj@rdesh26·Nov 17

A lot of voice AI releases these days are black boxes, so I appreciate the technical blog with details about different training stages. Of particular interest (and relevant to the recent buzz around "training voice models without text") is the following figure which shows the big jump in intelligence which comes from text-based planning.

Post attachment

This tweet is unavailable

Jensen Huang avatar
⟳ Desh Raj retweeted
Jensen Huang@JensenHuang·Nov 17

Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen

Faraz Siddiqi avatar
⟳ Desh Raj retweeted
Faraz Siddiqi@farazmsiddiqi·Nov 16

Here's the difference between speech-to-speech (S2S), cascaded, half-duplex, and full-duplex Voice AI models, and why the configurations matter.

Post attachment
Desh Raj avatar
Desh Raj@rdesh26·Nov 14

It has already worked for "actual large-scale runs". Ultra is a 550B-A55B model and uses nvfp4 for both forward and backward in pre-training.

Post attachment

When I first talked to @jackcookjack, he thought that although he was having fun working on nvfp4 pretraining in academia, he was skeptical that it would ever actually work for actual large-scale runs. As it turns out, he was wrong :^) This blog post explains how it can be done

Desh Raj avatar
Desh Raj@rdesh26·Nov 13

Very interesting work studying the placement of vision encoders on the multimodal performance. During JSALT 2026, we showed that encoder-free LLMs (like Gemma-4-12B and Inkling-Small) actually re-learn encoder-like properties in their early layers. "Encoder-like" means that: 1) the representations are invariant to cross-modal context; and 2) the representations for different layers show similar characteristics as those from self-supervised encoders. From this result, it naturally follows that bringing the encoder layers into the LLM backbone and reserving them for the specific modality (what Ellis calls "in-fusion") would provide the benefits of strong performance while still retaining an encoder-free architecture.

🔑 reserve them for vision same 0.4B params, same layers: → widen the shared FFNs: 40.1 → add a branch for all tokens: 42.3 → make that branch vision-only: 45.0 an encoder gets 44.4 [7/n]

0:08
Daebak 앤디 avatar
⟳ Desh Raj retweeted
Daebak 앤디@MyDaebakCafe·Nov 13

I am actually disgusted by the timing of this drop. And this will get a bit more technical. Let’s do a real technical teardown of this "1,385-sigma Blackwell weight fingerprint" report that OpenAI and Anthropic are breathlessly promoting today, because banking on nobody reading past the headline is how DC narratives get manufactured. ​First, look at the actual CUDA maths: when PyTorch initialises model weights (torch.nn.init) with a fixed PRNG seed, curand/Philox parallelises random number generation across the GPU’s Streaming Multiprocessors (SMs). Because Hopper has 132 SMs and Blackwell has 148 or 152 SMs, the exact same seed lays out initial weights in a different spatial pattern depending on the GPU’s SM count. In huge MoE models, certain low-gradient null-spaces or unrouted slices preserve faint directional traces of those step-0 weights even after training. ​So yes, brute-forcing the seed and SM grid can show that a model’s step_0 weight initialisation script ran on a 148/152-SM Blackwell GPU. Now read what the summary explicitly admits in the fine print: 1. Initialisation is not full training: The report literally states that these signatures show how the weights were created, and "support (but do not prove) the inference that Blackwell also participated in model training". Initialising a model checkpoint takes seconds on a single rented dev node before loading the safetensors onto a standard Hopper cluster. The "2,000 to 3,000 Blackwell GPUs" and "$500 million" cluster numbers do not come from the weight fingerprint and Depue admits in his own thread that he simply "estimated" cluster sizes from release schedules and model details. ​2. Zero geolocation and zero proof of smuggling: The final sentence of the executive summary explicitly admits that the findings "do not establish the physical location of the chips, how they were acquired, or whether export control regulations were violated". Depue himself rented a cloud GB200 instance just to run his verification test. Commercial labs that are not on the Entity List renting bare-metal Blackwell nodes in offshore clouds (Singapore, Japan, or the Middle East) is completely legal under BIS rules and leaves the exact same SM fingerprint. Now ask yourself why a former OpenAI researcher dropped this on November 12, the exact day Jensen Huang was pushed out, with OpenAI and Anthropic immediately stepping in to "independently verify" it. US closed-source labs are watching their paid API moats get eaten alive by free Chinese open-weights models. Conflating a step-0 PRNG initialisation seed with a "$500M smuggled Blackwell cluster" hands DC hawks the exact pretext OpenAI and Anthropic want to regulate global cloud compute and kneecap open-source AI.

I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.

Post attachment
Desh Raj avatar
Desh Raj@rdesh26·Nov 11

Very interesting post on "upstreaming" data to earlier training stages. As capabilities become more foundational, it makes sense to push them up the stack since that's where most learning happens. In the audio/speech case, for instance, fundamental capabilities like speech-text modality alignment should be learned in pretraining/CPT.

Notes on MAI, Beam, ML4 & Nemotron: pretraining(PT) can't just watch NLL, and CPT isn't just lr annealing. Inject multi-turn tool patterns in CPT to raise the pre-RL start. Part of my recent big-picture thoughts after 3 months at pretrain data team.