多信息融合的复杂动态场景高效感知与语义重建

Semantic understanding and 3D reconstruction of dynamic scenes have a wide range of potential applications in the fields of military, entertainment and intelligent transportation. Among the methods of scene perception and reconstruction, visual based method has attracted more and more attentions as it is of lower cost, universal and easy to use. However, the complexity and varying characteristics of dynamic scenes, especially outdoors scenes, bring great challenges for building efficient, robust and precise scene perception and semantic reconstruction systems through visual sensing devices only. To handle these challenges, this project plans to conduct a deep study on the key techniques involved in the image understanding and 3D scene reconstruction from the perspective of multiple cues fusion. By fully exploiting the complementarity and interaction among multiple perception cues and leveraging various scene priors and contexts, we aim to explore and propose theories, methodologies and their efficient implementation solutions for the interactive learning mechanisms of multiple scene perception tasks and the multiple context fused 3D reconstruction. More specifically, we will first investigate and propose an approach for joint perception of scene semantics, depth and disparity with a deep neural network. Then for dynamic scene reconstruction, we first propose to detect moving objects by integrating semantics and track them with a multiple deformable kernels based approach; then a motion sensitive dynamic scene reconstruction method is presented. At last, we will study and propose effective priors and context for the semantic segmentation of 3D volumetric data. We will further address the scalability, efficiency and robustness issues in the semantic reconstruction for large scale outdoors scenes. Our research will surely make an important advance in the theory of semantic reconstruction of dynamic scene and will undoubtedly support the applications of autonomous robot navigation, augmented reality and autonomous driving.

动态场景的语义理解和三维重建在军事、娱乐和智能交通等领域具有广泛应用前景，这其中基于视觉的感知和重建方式因具有低成本、普适易用等优势而备受关注。然而，动态场景特别是室外场景复杂多变，使得仅靠视觉感知设备来构建高效、鲁棒和精确的场景感知和重建系统面临着诸多挑战性问题。针对这些挑战，本项目拟从多信息融合角度，充分利用多感知信息的互补和协同性以及场景先验和上下文等知识，对二维图像理解和三维场景重建关键技术进行深入研究，探索并提出场景多感知任务协同学习机制和融合多种上下文信息的三维场景语义重建理论、方法及高效实现。具体包括：基于深度学习的场景语义、深度和视差联合感知方法，结合语义的运动对象检测和基于可变形多核模型的对象跟踪方法，运动感知的动态场景重建和多知识融合的三维数据语义分割方法。本项目将有力推动动态场景的语义重建理论发展，为机器人自主导航、增强现实、自动驾驶等应用提供重要技术支撑。

复杂动态场景的三维感知、重建和语义理解是实现很多智能应用如无人系统导航及作业、公共安全视频监控、军事目标侦察识别等应用的核心技术，也是我国《新一代人工智能发展规划》中亟需重点突破的共性关键技术。本项目围绕其中的关键问题进行了较深入研究，取得了一系列创新性研究成果。在场景深度超分辨率恢复方面，分别提出了基于边界学习和边界引导插值的深度图像超分辨率方法、渐进式多分支聚合的场景深度超分辨率网络、基于深度范围切分的深度图像超分辨率网络和基于跨任务知识迁移的深度图像超分辨率方法等；在双目视差估计方面，研究并提出了基于几何与上下文关系的两阶段立体匹配方法、基于成本亲和力学习网络的立体匹配方法、平衡深度预测的立体匹配网络等；目标检测方面，提出了尺度自适应的目标检测方法、单目图像3D目标检测的数据表示和检测方法等；在场景三维重建方面，融合特征SLAM和深度估计网络实现了大场景的稠密重建，提出了标志辅助的多特征融合定位与建图方法；在场景语义分割方面，提出了基于视差图与地平面拟合的地形语义分割和跨类别的语义对象分割方法；在场景语义识别方面，对场景图像细粒度识别和跨域识别进行了深入研究，分别提出了基于关系挖掘的判别性特征抽取与表征方法、基于类别语义一致性学习网络的细粒度识别、基于动态位置敏感学习网络的细粒度识别等方法，以及基于置信度正则标签传播的领域自适应方法、源领域稀疏标注的弱监督领域自适应方法等。此外，基于项目成果研发的多目多波段立体感知设备获取2019年全军实物招标第一名，目前已装备我军无人战车，为服务国家重大需求做出了重要贡献。所研发关键技术助力团队水下机器人获得水下机器人大赛自主抓取第一名。以上成果发表论文30篇，其中期刊论文11篇，会议论文19篇，专利9项。项目成果有力推动了三维感知、重建和语义理解的理论和技术研究，对相关应用系统具有重要借鉴意义。