基于深度与图像信息融合的场景理解及应用

Scene understanding has wide applications in many fields, ranging from digital city construction, robot navigation, autonomous driving to virtual reality. In spite of numerous efforts in the past decade, understanding complex scenes remains an unsolved challenge due to lacking efficient 3D reconstruction method. This project proposes a new method towards understanding full 3D natural scenes in real time. It relies essentially on fusing depth and visual image cues. We first extract basic features that are discriminative and highly robust to lighting and occlusions to match with each other. During the course of matching, we integrate middle-level contexts and adaptive priors to improve the matching rate. By viewing feature matching as a constrained quadratic programming problem, natural scenes are efficiently registered and 3D pose is correctly estimated. Following the results, we utilize machine learning tools to recognize terrain, rely on 3D geometry inference to detect obstacles and recognize scenes via 3D scene registration, which can effectively solve the challenging problems of foothold selection in a walking robot, obstacle avoiding in swarm robotics and SLAM in a mobile robot.

场景理解是一个在数字城市构建、机器人导航、无人驾驶和虚拟现实等诸多领域有着广泛应用的重要研究方向。虽然有着多年的研究，但由于三维重建技术的不成熟，解析复杂的场景仍是一个难点问题。为此，本项目提出了一种基于深度和图像信息融合的三维场景理解的新方法。利用深度图像与可见光图像的互补性，定义了判别性高且对光照变化、遮挡等干扰信号鲁棒的三维低层基元特征；在匹配过程中，加入了中层语义信息和可变先验模型约束，并通过求解一个有约束的二次规划问题实现三维场景匹配和姿态估计，最终实现完整的场景重建。在此基础上，利用分类学习的方法识别地形、根据几何信息的推理检测障碍物和匹配三维场景实现场景识别，有效地解决了四足机器人平稳落足点的选择、群机器人的自主避障和移动机器人的自主导航和定位等难点问题。

随着RGB-D摄像头的普及，深度信息的获取变得更加容易，这大大缓解了计算机视觉尤其是三维视觉经典问题的困难程度，为此，本课题通过融合深度和RGB颜色信息，对三维视觉问题进行了重新解析，希望探求出新的视觉理论方法和工具。在过去的三年中，该课题主要针对三维姿态跟踪、三维语义信息提取、深度信息修复、运动去模糊和深度人体行为识别等子问题进行了深入系统地研究，累计发表文章13篇，包括SCI期刊文章4篇，其中，2篇发表在视频处理顶级期刊IEEE TCSVT上，1篇发表在IEEE JBHI上。另外，在ICPR、ICIP、ICASSP和ACPR等国际会议上累计发表文章9篇。该项目所取得的研究成果，不仅推动了该领域相关理论的进展，也解决了很多实际的工程问题，取得了很好的学术和工程成果。