
My way of contributing to PyTorch since I joined: complain and whine until something happens :D I was super happy by PyTorch team's reaction on this one! I complained about their current RNG API. I think it's horrible. I made a prototype of what I think it should look like, and we used that in our research codebase; but the implementation was horrible hacks on top of the current API. Then, I wrote a big doc explaining what I think the design should be like, and why, and how the current API simply doesn't allow it. I didn't hear much for a little while, but then they came back with a whole new design, largely the way I wanted it, plus some more. We iterated a little more on the rough edges of that (we iterated == I try it and complain), and here we are. (This is just my perspective, I'm sure there were others also requesting something like it, that I didn't see.) It's inspired by both NumPy's new API and JAX's API, but it also goes beyond both. I really like it; if you embrace the API, it gives you full control and awareness of PRNG in your codebase. It even forces you to think and make decisions you wouldn't have realized that you otherwise made implicitly, until they come back and bite you! This makes me think, maybe I should write a little guide/example of how to use it in some toy training codebase some time...
We have an experimental (currently private, but we'll soon be promoting it to public) stateless RNG API in PyTorch that does not have this problem, you should try it! github.com/pytorch/pytorch/b… 🧵






