强化学习
概率逻辑
控制器(灌溉)
数学优化
计算机科学
趋同(经济学)
航程(航空)
弹道
增广拉格朗日法
控制理论(社会学)
分歧(语言学)
数学
控制(管理)
人工智能
工程类
航空航天工程
天文
农学
经济
哲学
物理
生物
经济增长
语言学
作者
Baiyu Peng,Jingliang Duan,Jianyu Chen,Shengbo Eben Li,Genjin Xie,Congsheng Zhang,Yang Guan,Yao Mu,Enxin Sun
标识
DOI:10.1109/tnnls.2022.3175595
摘要
Safety is essential for reinforcement learning (RL) applied in the real world. Adding chance constraints (or probabilistic constraints) is a suitable way to enhance RL safety under uncertainty. Existing chance-constrained RL methods, such as the penalty methods and the Lagrangian methods, either exhibit periodic oscillations or learn an overconservative or unsafe policy. In this article, we address these shortcomings by proposing a separated proportional-integral Lagrangian (SPIL) algorithm. We first review the constrained policy optimization process from a feedback control perspective, which regards the penalty weight as the control input and the safe probability as the control output. Based on this, the penalty method is formulated as a proportional controller, and the Lagrangian method is formulated as an integral controller. We then unify them and present a proportional-integral Lagrangian method to get both their merits with an integral separation technique to limit the integral value to a reasonable range. To accelerate training, the gradient of safe probability is computed in a model-based manner. The convergence of the overall algorithm is analyzed. We demonstrate that our method can reduce the oscillations and conservatism of RL policy in a car-following simulation. To prove its practicality, we also apply our method to a real-world mobile robot navigation task, where our robot successfully avoids a moving obstacle with highly uncertain or even aggressive behaviors.
科研通智能强力驱动
Strongly Powered by AbleSci AI