←
Edward Z. Yang9,258 posts
Edward Z. Yang avatar

Edward Z. Yang

@ezyang

I work on PyTorch at Meta. Chatty alt at @difficultyang.

◷ Joined May 2008▧ 175 Photos and videos
Lucas Beyer (bl16) avatar
⟳ Edward Z. Yang retweeted
Lucas Beyer (bl16)@giffmana·Nov 16

My way of contributing to PyTorch since I joined: complain and whine until something happens :D I was super happy by PyTorch team's reaction on this one! I complained about their current RNG API. I think it's horrible. I made a prototype of what I think it should look like, and we used that in our research codebase; but the implementation was horrible hacks on top of the current API. Then, I wrote a big doc explaining what I think the design should be like, and why, and how the current API simply doesn't allow it. I didn't hear much for a little while, but then they came back with a whole new design, largely the way I wanted it, plus some more. We iterated a little more on the rough edges of that (we iterated == I try it and complain), and here we are. (This is just my perspective, I'm sure there were others also requesting something like it, that I didn't see.) It's inspired by both NumPy's new API and JAX's API, but it also goes beyond both. I really like it; if you embrace the API, it gives you full control and awareness of PRNG in your codebase. It even forces you to think and make decisions you wouldn't have realized that you otherwise made implicitly, until they come back and bite you! This makes me think, maybe I should write a little guide/example of how to use it in some toy training codebase some time...

We have an experimental (currently private, but we'll soon be promoting it to public) stateless RNG API in PyTorch that does not have this problem, you should try it! github.com/pytorch/pytorch/b… 🧵

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

Stateless RNG is really good when you need to do careful management of your randomness, and it's also SM-independent so no more fingerprinting of your GPU types too :) Give it a try. FIN.

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

So in the end, I think we ended up never fixing it upstream, and people did a lot of bodgy things in userland. But @giffmana finally got us to pay attention to stateless RNG for other reasons, jbschlosser finally implemented this API, based off of the JAX APIs.

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

When Wei originally proposed how we would fix it, it involved the insanity of very carefully working out how exactly the sampling was done. It's also a bit tricky even if you're willing to do a one time BC break how how the RNG is generated; ideally user code changes too.

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

Originally when this problem was posed to me, I thought, "no problem, we just have to make sure we advance the RNG stream appropriately for each rank and it will work." NOPE. Because the exact assignment of Philox to tensors is extremely sensitive to exact hardware details.

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

Suppose you have a model that runs on TP=1 or TP=2. In the TP=2, the weight has been split across two separate nodes. Initializing the TP=1 case is a single RNG kernel call, whereas in TP=2 you are you have two RNG calls on different nodes. You want the result to be the same!

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

I've actually been dimly aware of the RNG SM-dependence problem for some time, because we ran into it in a different setting: how can you ensure that your parameter initialization is invariant to the choice of parallelism?

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 16

We have an experimental (currently private, but we'll soon be promoting it to public) stateless RNG API in PyTorch that does not have this problem, you should try it! github.com/pytorch/pytorch/b… 🧵

I have a major result to share. I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China. This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model. Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves. I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems. I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used. It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20. These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs. I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at unemploymentlabs.com/checkpo…. There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.

Post attachment
PyTorch avatar
⟳ Edward Z. Yang retweeted
PyTorch@PyTorch·Nov 11

What can you test on one GPU before a big PyTorch training run? PyTorch core maintainer @ezyang (@Meta) explains in this clip from #PyTorchCon North America 2026. He describes how a fake process group lets developers stub out distributed communication and check for crashes and GPU out-of-memory errors. He also discusses using SPMD types to look for numerical bugs without running a full-scale distributed test. In the full keynote, PyTorch core maintainer @albanDesmaison (@Meta) discusses hardware backend integration and testing. @ezyang also covers activation checkpointing with torch_remat and discusses FlexShard. Watch the full keynote: youtu.be/EwZafo8QDDE?si=-Y2b…

2:36
Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

It is not the end of the world if FV is only used for simple verification conditions like "no crashes". But the most interesting theorems are about humans trusting software, and here we essentially have a problem of alignment: the human cannot be removed from this loop. FIN.

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

If you have a big pile of vibe coded Python scripts that elaborate a trace of your Python program into Lean, how much should you actually trust this TCB? And if you don't trust this, are you going to write a formal semantics of Python? Will you trust that? (cough Starlark)

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

In my alt, I recently joked that Python would survive because humans like it. But this is in fact a serious problem. If your users demand specs written in Python, how exactly are you going to choose to bridge the formalization gap?

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

Even in a "nice" specification language, the temptation to use more features in your specification language, or have your specification be generated by some sort of trusted elaboration/metaprogramming mechanism is high. But it also increases the risk you make a mistake in spec!

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

Similarly, C compilers famously have non-bug behavior where they can behave counterintuitively with respect to what actually runs at runtime: e.g., UB after a condition allows the compiler to assume the condition must always be false. Better prove your spec has no UB!

Edward Z. Yang avatar
Edward Z. Yang@ezyang·Nov 9

However, we must then ask the question: is your programming language actually a good specification language? We wish for a speclang to have a small TCB, but if you rely on a low level semantics on the compiled output, then correct compilation of your spec is in your TCB!