基于多梯度递推方法的强化学习多智能体系统跟踪控制问题研究

Reinforcement learning does not rely on the accurate environmental model and complete prior knowledge, which can improve the decision and action ability of multi-agent systems in complex environments. As a consequence, it is of great significance to study the multi-agent systems control based on reinforcement learning. In this project, for uncertain nonlinear multi-agent systems, the tracking control algorithm which is based on the reinforcement learning approach will be proposed. A series of problems will be settled including local optimal, precision tracking, and network attack, along with the effectiveness of the proposed scheme validated by theoretical analysis and semi-real simulation. The research mainly includes: 1. For a class of strict feedback multi-agent systems, a multi-gradient recursive (MGR) reinforcement learning strategy will be proposed to deal with the local optimal problem and improve the energy conservation. 2. The sufficient condition of accuracy tracking will be proposed based on the output regulation feedback signal, which will overcome the inaccuracy of uniform bounded convergence. 3. With regard to the security problem of multi-agent systems, the attack detection and prevention strategies, which are based on locally weighted learning and reinforcement learning, respectively, will be proposed for multi-agent systems, and thus the system information security problem will be addressed. Finally, the results will be applied to cooperative control for the automated vehicle and aircraft. The main objective of this project is to propose a more complete set of multi-agent systems tracking control theory based on reinforcement learning, and in turn, provide practical design method for engineering and technical personnel.

强化学习不依赖精确的环境模型和完备的先验知识，能够提高多智能体系统在复杂环境中的决策和行为能力，因此研究强化学习多智能体系统控制问题具有重要意义。本项目针对不确定非线性多智能体系统，提出基于强化学习的多智能体系统跟踪控制算法，解决局部最优、精确跟踪以及网络攻击等问题，通过理论分析和半实物仿真验证算法的有效性。主要研究内容包括：1.针对一类严格反馈的多智能体系统，提出一种分布式多梯度递推的强化学习控制算法，解决控制系统局部最优问题和节能问题；2.基于输出调节理论，设计反馈补偿信号，提出精确跟踪充分条件，解决一致有界收敛不精确问题；3.针对伴随攻击的多智能体系统，提出基于局部加权学习的攻击检测和强化学习的攻击防御控制策略，解决系统信息安全问题。最后，将所研究的成果应用于无人车和无人机地空协同控制系统。提出一套较为完整的强化学习多智能体系统跟踪控制理论，并为工程技术人员提供实用的控制设计方法。

随着计算机技术、人工智能理论以及系统信息安全科学的深入发展，以及伴随着全球信息化的高度融合，对自动控制的要求变得越来越高。多智能体系统能够解决单一个体难以实现的目标并且能够满足日益复杂的综合任务需求。然而，多智能体系统的不确定性、强非线性、高度开放性的特点为多智能体系统协同控制带来一定的挑战性。本项目针对不确定非线性多智能体系统，基于强化学习方法开展协同控制方面的研究，取得了三方面的研究成果：（1）研究了强化学习中的局部最优问题。采用重复学习过去时刻的信息，构造一种新型的伪梯度，提出了一种多梯度递推的强化学习算法，能够保证参数估计收敛到真值；（2）考虑系统收敛精度问题，提出一种指定性能的强化学习方法，通过系统转换将受约束的系统误差转换为一种新的不受约束的误差变量，保证系统跟踪误差收敛到指定的控制精度；（3）考虑到控制系统的安全性问题，采用局部加权学习方法建立被控对象高精度运动数学模型预报被控制对象运动状态，通过对比运动状态与预报值判断被控对象是否受到攻击。上述基于强化学习方法设计的协同控制器，能够通过增强反馈信号实现系统与环境之间的交互，降低了系统对被控对象模型的依赖，同时降低了执行器的更新频率和参数初值的依赖。近三年发表学术论文15篇，包括IEEE汇刊5篇，授权发明专利2项，荣获国际会议最佳论文奖1项。