←
Zikang Xiong12 posts
Zikang Xiong avatar✓

Zikang Xiong ✓

@caffe_tt

Autonomous System

◷ Joined December 2016▧ 2 Photos and videos
Wei ZHAN avatar
⟳ Zikang Xiong retweeted
Wei ZHAN@Wei_ZHAN_·Oct 27

Can we achieve state-of-the-art for mainstream E2E and vectorized driving benchmarks without using any human demonstration data? Terra series at @AppliedInt now tops the NAVSIM, HUGSim, nuPlan val14 and InterPlan benchmarks, powered by self-play RL without imitation, scaling in closed loop with all agents being reactive. 🚘 With TerraZero released a few months ago, we set the SOTA on vectorized benchmarks like nuPlan val14 and InterPlan, powered by our super high-throughput self-play RL framework experiencing 25 centuries of challenging driving scenarios. 📽️ In June this year, TerraTransfer was released as the first of its kind to bring the SOTA vectorized policy from self play (TerraZero) to an E2E policy via image-to-vector encoder alignment, which topped the HUGSim closed-loop score. 🔚 This week, we are thrilled to release TerraTransfer++, a demonstration-free E2E planner setting the SOTA for NAVSIM and HUGSim, by adding planning capability from self-play rollout experience to TerraTransfer. Setting SOTA for public benchmarks is just a beginning, as we have seen the journey from pioneering work like UniAD to successful mass production of E2E ADAS in the industry, but what makes me quite excited is to see the strong signal from a new paradigm with potential to refactorize how the next generation autonomy can be developed, and bring the scalability, safety and performance to the next level. We are the believer of data and scaling, but not just human demonstration data and open-loop scaling like imitation or open-loop RL (AV2.0) - we have the bet on closed-loop scaling with both synthetic and real-world data (AV3.0), and the Terra series is the starting point of the journey towards real product with our bet. Great work from the team @caffe_tt, Weixin Li, Zhouchonghao Wu, @AkshayRangesh, Saarth Bonde, Grantland Hall, @ChenTangMark, Yihan Hu, for creating TerraTransfer and TerraTransfer++! TerraTransfer++ report:  arxiv.org/abs/2610.30319 TerraTransfer++ website: terra-applied.github.io/Terr… TerraTransfer report: arxiv.org/abs/2606.17386

1:17
Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(8/8) Finally, the generated Lyapunov function is further used to build a region of attraction providing explicit safety enhancement instead of explicit enhancement provided by reward shaping or constrained optimization.

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(7/8) We co-learn the neural Lyapunov function and control policy into a reinforcement learning loop. The neural Lyapunov function can adapt to the control policy, while it also acts as a critic and provides an additional training signal for the actor (control policy).

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(6/8) Although nowadays constrained reinforcement learning algorithms like CPO can better handle safety constraints, it is still very challenging for them to provide reasonable safety enhancement when it comes to cluttered environments, as shown in our work.

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(5/8) Small coefficients may make the trained agent be reckless and ignore the safety reward. In contrast, large coefficients may make the agent be very conservative, and stay in place, since no collision can provide a decent cumulative reward.

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(4/8) Hacking/shaping rewards may provide better results. For example, adding the distance to the unsafe region into the reward function may play some role. However, the proportion/coefficient of such safety reward terms is a problem.

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(3/8) Safely navigating a robot in cluttered environments is challenging, especially when the robot dynamics are complex and unknown. Deep reinforcement learning, as one strong candidate for such a problem, is typically hard to provide trustful safe behaviors.

Zikang Xiong avatar
Zikang Xiong@caffe_tt·30 Jun 2022

(2/8) Many thanks to my advisors Ahmed H. Qureshi (@ahquresh) and Suresh Jagannathan, as well as my co-author Joe Eappen (@yokian). My coding buddy, Armstrong, also deserves his favorite salmon treat.

Post attachment
C-BRIC avatar
⟳ Zikang Xiong retweeted
C-BRIC@CBRIC_Center·7 Feb 2022

C-BRIC researchers and partners can join Zikang Xiong for a talk on "HiSaRL: A Hierarchical Framework for Safe Reinforcement Learning" Thurs., Feb. 10 at 11:00am ET/8:00am PT.

Post attachment