基于深度对抗网络和强化学习的遥感视频多目标检测与跟踪研究

It is difficult to identify the small-sized objects due to complex and varied scenes, and track the lost objects due to inter-frame geometry and radiation difference, and object occlusion in the satellite remote sensing videos. In order to solve these difficulties, fully convolutional-based generative adversarial network model is constructed. It improves the performance of semantic segmentation for remote sensing videos by adversarial learning between the generative and discriminative networks, which provides semantic priori for subsequent object detection and tracking. The position-sensitive strategies are designed for classification and location. The region-based fully convolutional network model is constructed with the embedded semantic priori. Meanwhile, the complementary information of temporal context in the remote sensing videos is used. These methods improve the detection accuracy of small-sized objects and implement end-to-end multi-object detection. Combining with powerful representation ability of deep learning and efficient decision-making mechanism of reinforcement learning, a new multi-object tracking method based on deep reinforcement learning is devised. It tracks of occluded objects again and obtains the complete moving trajectories of objects by using the data correlation of existing object trajectories and detection results. Finally, the project plans to achieve accurate, efficient and robust multi-object detection and tracking of remote sensing videos. The Skysat and Jilin-1 satellite remote video datasets in the complex background are used to validate the effectiveness and superiority of the new model and algorithm. These studies are expected to accelerate the development of theory and technology in remote sensing video interpretation, and promote the application of remote sensing videos.

卫星遥感视频存在场景复杂多样、目标所占像素少引起的目标难以辨别及目标帧间几何辐射差异大、目标遮挡引起的目标丢失难以跟踪的难题。针对这些难题，本课题拟建立全卷积生成对抗网络模型，通过生成网络和判别网络的对抗学习提升其遥感视频语义分割性能，为后续目标检测与跟踪提供语义信息；设计位置敏感的分类和定位策略，构建语义嵌入的区域全卷积网络，同时利用遥感视频的时序上下文互补信息，提高对小目标的检测精度，实现端对端的视频多目标检测；结合深度学习强大的表征能力和强化学习高效的决策机制，设计深度强化学习多目标跟踪方法，利用目标已有轨迹与检测结果的数据关联，实现被遮挡目标的再次跟踪，获取目标的完整运动轨迹，最终达到准确、高效、鲁棒的遥感视频多目标检测与跟踪目的。通过复杂背景下Skysat和吉林一号卫星视频数据验证新模型和算法的有效性和先进性。这些研究有望推动遥感视频解译理论和技术的发展，促进遥感视频应用的深入。

针对遥感视频因目标所占像素少、几何和纹理特征匮乏引起的目标难以辨别及帧间目标几何、辐射差异大、目标遮挡引起的目标丢失难以跟踪的问题，本课题面向卫星遥感视频多目标检测与跟踪任务，围绕遥感视频语义分割、感兴趣目标检测、多目标跟踪，开展了以下工作：1）建立基于全卷积的生成式对抗网络模型，设计两个生成器来生成分别包含空间信息和光谱信息的样本，利用判别器来提取遥感图像的空谱特征同时输出分类置信度，实现高效精准的语义分割，为后续的目标检测与跟踪任务提供了辅助；2）构建基于帧间信息融合与错误匹配抑制的遥感视频运动目标检测网络模型，提出一种基于融合帧间信息的关键点目标检测网络，主要解决遥感视频中小尺寸和模糊目标的误检与漏检问题；3）提出了一种融合运动信息的自适应锚点R-CNN遥感视频目标检测方法。通过均值差分方法从遥感视频中提取目标运动信息，并将其与表观信息相融合，获得具有辨别力的目标特征；对于小尺寸目标，通过定义平滑因子来改进损失函数，使得目标尺寸的回归性能更加稳定；4）设计基于双LSTM分支的遥感视频多目标跟踪方法，其中一个LSTM分支，用于提取同一帧内相邻目标之间的空间信息，有利于稳定地跟踪缺乏表观信息的密集车辆；另一个LSTM分支根据连续帧的运动速度建立车辆的运动模型。基于时空信息的多目标跟踪网络通过结合运动信息和空间信息来判断假设轨迹是否为真，同时为丢失或遮挡目标回归一个虚拟位置。即使一些车辆在几帧中丢失或被遮挡，当这些车辆再次出现时，跟踪可以有效地将这些车辆与原始轨迹相关联。在项目资助下，项目组发表学术论文34篇，其中SCI检索期刊论文25篇；申请国家发明专利12项，其中授权3项。项目组培养已毕业硕士生6名。项目负责人依托该项目，荣获中国自动化学会自然科学二等奖，入选“中国科协青年托举工程计划”和“陕西省科技新星”。