面向无人机集群时敏目标搜索的深度强化学习及云端训练优化

For the needs of large-scale time-sensitive target search in the fields of disaster response, environmental protection, etc., this project focuses on the problem of the time-sensitive target search for UAV swarms under uncertain conditions, based on the multi-agent deep reinforcement learning and cloud computing distributed parallel technology. We will research the following three aspects: 1) In view of the uncertainty of target distribution, sensory observation and other uncertain factors affecting the effectiveness of UAV swarms cooperative search in the real application, the modeling method for the time -sensitive target search for UAV swarms under uncertain conditions is studied. 2) According to the characteristics of the non-stationary environment of multi-agent system, combined with the characteristics of neighbor cooperation in UAV swarms, we study the multi-agent deep reinforcement learning algorithm based on neighbor interaction under non-stationary environments. 3) For the fact that deep reinforcement learning relying on a large number of environmental interactions leads to long training time, by using distributed parallel technology in cloud computing, we design a distributed storage architecture for empirical data, and research an asynchronous parallel training optimization method for multi-agent reinforcement learning. This project aims to explore practical and feasible approach for the time-sensitive target search for UAV swarms from the perspective of intelligent learning and effectively improve the efficiency of large-scale time-sensitive target search. This project will provide important supports to the research and application of UAV swarms and new generation artificial intelligence.

面向灾难响应、环境保护等领域中大范围时敏目标搜索的重要需求，本项目基于多智能体深度强化学习和云计算分布式并行技术，围绕不确定条件下无人机集群时敏目标搜索问题，展开三个方面的研究：1）针对现实应用中目标信息、传感器观测等不确定因素对方法有效性造成严重影响的问题，研究不确定条件下无人机集群时敏目标搜索的感知决策建模方法；2）针对多智能体系统非稳态环境的特点，结合无人机集群邻居局部协作的特征，研究非稳态环境中基于邻居交互的多智能体深度强化学习算法；3）针对深度强化学习依赖于大量环境交互导致训练耗时长的问题，结合云计算分布式并行技术，设计经验数据的分布式存储架构，研究多智能体深度强化学习的异步并行训练优化方法。本课题旨在从智能学习的角度，探索切实可行的无人机集群时敏目标搜索方法，有效提高大范围时敏目标搜索的效率，并为无人机集群及新一代人工智能的理论研究和应用提供支持。

面向灾难响应、环境保护等领域中大范围时敏目标搜索的迫切需求，本课题基于多智能体深度强化学习和云计算分布式并行技术，围绕无人机集群时敏目标搜索问题，展开了三个方面的研究：1）针对现实应用中目标分布、传感器观测等不确定因素对方法有效性造成严重影响的问题，研究了不确定条件下无人机集群时敏目标搜索的感知决策建模方法；2）针对多智能体系统非稳态环境的特点，结合无人机集群邻居局部协作的特征，研究了基于邻居交互的多智能体深度强化学习算法；3）针对深度强化学习依赖于大量环境交互导致训练耗时长的问题，结合云计算分布式并行技术，研究了多智能体深度强化学习的异步并行训练优化方法。项目提出了无人机集群时敏目标搜索序贯决策模型、面向无人机集群时敏目标搜索的多智能体强化学习算法、基于云计算的异步并行优化方法等模型及算法，并对所提方法进行了仿真验证，与现有方法相比，所提方法使得无人机集群搜索跟踪到的目标数量分别高出26% 、12% 和17%，同时可以扩展到大规模（1000架）集群中。所提的基于邻居交互的无人机集群碰撞规避算法，能够有效实现无人机集群在飞行过程中的碰撞规避。所设计的基于云计算的异步并行优化方法，运用3台服务器，能够有效的提高训练速度，与1台服务器相比，加速比最高可达到8.29。项目成果发表了学术论文13篇，其中，CCF/CAA推荐A类或B类期刊7篇，1篇1作论文获全球ESI前1%高被引，1篇论文获入选《航空学报》2020年最受关注论文。项目成果为基于学习的无人机集群决策控制方法奠定了基础，推动了无人机集群和新一代人工智能的理论研究及应用拓展。