基于卷积神经网络的深度图像超分辨率重建及空洞修复技术研究

The depth image is able to represent the 3-dimensional spacial position of an object. It has been widely used in many hot research fields, such as self-driving car, virtual reality, human-computer interaction, etc. But the common low-resolution and depth hole artifacts are nonnegligible quality-affecting factors, and how to effectively improve the whole quality of the depth image has emerged as an urgent problem. Most existing depth image super-resolution and depth inpainting methods have some deficiencies in different aspects like pertinence, flexibility, and completeness, which are targeted as the main focus of this research. Based on our multi-scale symmetric network, we will follow the particular textureless and sharp edge characteristics of depth images and design a depth image super-resolution framework using dynamic selection and enhancement of guidance features. We will develop a generative-adversarial-networks-based depth inpainting method, which combines a content structure generative network, a detail generative network, and a PatchGAN discriminator to flexibly deal with irregular depth holes in depth images. We will also integrate the super-resolution result with the depth inpainting result by exploring the attention mechanism and conditional random field model to achieve multi-source fusion and multi-layer fusion, respectively. The final purpose of this proposal is to improve the depth image quality and further expand its applications.

深度图像反映了目标在空间中的三维位置信息，它在自动驾驶、虚拟现实、人机交互等热门研究领域都具有广泛的应用价值。而深度图像中常见的低分辨率和空洞缺陷不容忽视，如何有效地提高深度图像的整体质量是一个亟待解决的问题。已有的大多数深度图像超分辨率重建及空洞修复算法在针对性、灵活性和完整性方面尚存不足，本项目拟以此为着力点进行深入的研究。本项目拟在多尺度对称网络结构的基础上，针对深度图像纹理少、边缘锐化的特征，构建基于引导特征动态选取与增强的深度图像超分辨率重建框架。拟提出一种基于生成对抗网络的深度图像空洞修复算法，联合内容结构生成网络、细节生成网络和基于PatchGAN的判决网络，来灵活地处理深度图像中的不规则空洞。拟完善超分辨率重建与空洞修复的融合处理，探索使用注意力机制和条件随机场模型，以达到多源融合和多层融合的效果。本项目最终目的在于提高深度图像的整体质量，拓展深度图像的应用前景。

深度图像在自动驾驶、虚拟现实、人机交互等领域具有广泛的应用价值，各应用领域对深度图像的质量具有较高的要求。但受深度相机硬件条件限制，深度图像常存在着低分辨率和空洞缺陷。.本项目致力于研究深度图像超分辨率重建及空洞修复技术，以提高深度图像整体质量，主要工作包括：（1）研究了基于彩色引导特征动态选取与增强的深度图像超分辨率重建对称网络，有效抑制纹理过度转移和深度渗透瑕疵；(2) 对非局部均值约束在深度图像超分辨率重建中的应用进行了研究，有效抑制深度噪声；(3) 对基于深监督跨尺度注意力的深度图像超分辨率重建模型进行了研究，提升了重建网络的学习能力；(4) 研究了基于平行自适应修复网络的大空洞修复算法，以生成对抗网络为基本框架，联合内容结构分支网络、细节纹理分支网络和PatchGAN判决网络，提升了模型对大面积缺失区域的修复能力；(5) 对深度修复网络的多层次特征融合进行了研究，通过门控循环融合模块隐态的更新逐步融合低层次特征和高层次特征，提升了深度修复性能。.本项目有效提升了深度图像的整体质量，加速了深度图像的应用进程，拓展了其应用前景。