
when you argue with me about control theory this is who you’re arguing with

Full of childlike wonder. Teaching robots manners. RL Lead @ Apptronik. UT Austin PhD candidate. Past: Boston Dynamics AI Institute, NASA JPL, MIT ‘20.

when you argue with me about control theory this is who you’re arguing with

Big uncomfortable facts courtesy of @KyleMorgenstein
people kept asking me for “hot takes” at CoRL. I was sorry to disappoint, because what I came up with felt lukewarm: RL is the only thing that has ever worked at scale. exploration + adaptation. that’s it. everything else is a party trick. RL is the ultimate bitter lesson.

the main take away here isn’t that RL can do thinks classical control can’t, it’s that RL can do these things under degrees of model uncertainty that would make classical controllers shit the bed.

under a nominal model yes, but the core issue here is not the optimal policy, it’s the uncertainty over the model! you could write a robust controller with bounded uncertainty, but there’s a lot of structure and assumptions baked in that RL doesn’t require.
Did they really just throw reinforcement learning at the fucking inverted pendulum/two wheel robot stabilization problem? That's like the most common control system design project ever, classical control theory will get you something much more stable and entirely predictable🤦♂️

I agree with @KyleMorgenstein that a core difference between supervised learning and RL is that in imitation learning, we are approximating a function that generates behaviors like the demonstrations (hopefully optimal ones for some task!), whereas in reinforcement learning, you have to discover a function that would generate optimal behavior for some task. What I think is interesting is there a real spectrum of methods across RL and SL that make this point very salient. For example, while PPO and SAC are clearly RL, and behavior cloning is clearly SL, there are approaches that seem to have a blend of both: upsidedown reinforcement learning(UDRL)/advantage-conditioned policies, and advantage-weighted regression (AWR) come to mind. In UDRL, the policy can discover new behaviors without priors, but it trains like goal-conditioned self-supervised learning(where your goals are returns/advantage/etc.). AWR is like weighted behavior cloning, but that has deep ties to scored-based policy gradient methods like REINFORCE and PPO.
this isn’t some big secret. these are the first slides of my RL lecture notes that I teach at UT. function approximation is fine if you already know what you want, but intelligence is about function discovery. if you can write down a (proxy) objective, you can optimize it.

this isn’t some big secret. these are the first slides of my RL lecture notes that I teach at UT. function approximation is fine if you already know what you want, but intelligence is about function discovery. if you can write down a (proxy) objective, you can optimize it.

that’s a fine way to bootstrap priors, but if you want to do something interesting with them, you will eventually be doing RL.

“supervised vs self supervised” is a false dichotomy. you’re still training against some reference, even if that reference is a latent property of the dataset (eg contrastive methods).

when we talk about intelligence, we’re talking about RL. all other learning paradigms are shortcuts and hacks.

people kept asking me for “hot takes” at CoRL. I was sorry to disappoint, because what I came up with felt lukewarm: RL is the only thing that has ever worked at scale. exploration + adaptation. that’s it. everything else is a party trick. RL is the ultimate bitter lesson.
you may think “RL is not the right tool for this problem” and you would be wrong. it’s all RL. this is sick.

RL stonks 📈📈📈


you may think “RL is not the right tool for this problem” and you would be wrong. it’s all RL. this is sick.

We had fun in the sim2real trenches!
It also takes an amazing team of engineers like @_ildsjel @castillom90 @shubhamsingh091 @DhruvThank (+ many more not on Twitter) to do the hard work to make it right.

Qiayuan (@qiayuanliao) and I wrote about modeling the Unitree G1’s closed-loop ankle linkage in @mujoco. We look at how the linkage changes the ankle’s motion and dynamics, and what it takes to simulate it reliably. We spent quite a bit of time on the blog post making interactive demos and shareable models, and we hope you find it useful! kzakka.com/closing-the-loop/

It also takes an amazing team of engineers like @_ildsjel @castillom90 @shubhamsingh091 @DhruvThank (+ many more not on Twitter) to do the hard work to make it right.