面向三维数据的深度神经网络研究与设计

In recent years, significant breakthrough has been achieved on image based vision tasks, which greatly improved computer’s capability in understanding real world. However, images are inherently short of describing 3D structure of real world. It is foreseeable that, combined with 3D data, vision systems can understand real world more effectively, and thus interact and operate more accurately. Deep learning is the core technology for breakthroughs in image understanding in recent years and is considered to have great potential for processing 3D data. However, 3D data has characteristics that are quite different from images, and the attempt of porting image-oriented deep neural networks to process 3D data is not successful. This project aims at designing efficient and effective neural networks by leveraging the sparseness of 3D data, for applications such as object detection, recognition and segmentation, as well as 3D geometric modeling, which are the pillar technologies in computer graphics and computer vision. The key research issues to be addressed include deep learning-oriented 3D data representation, 3D sparse convolution operator design as well as high-resolution and high-diversity 3D model synthesis. The project has great potential in producing efficient and effective deep neural networks for 3D data, and generalizing the success of deep learning in processing image data to the processing of 3D data.

近年来，基于图片的视觉任务取得了突破性进展，极大提高了计算机理解现实世界的能力。然而，图片内在地缺乏对现实世界的三维结构描述。可以预见，结合三维数据，视觉系统能够更有效地理解现实世界，进而更准确地交互和操作。深度学习是近年来图片理解取得突破性进展的核心技术，也被认为有巨大的潜力适用于处理三维数据。然而，三维数据具有非常不同于图片数据的结构特征，将适用于处理图片数据的深度神经网络拓展并用于处理三维数据的尝试并不成功。本项目研究如何有效利用三维数据在空间上稀疏的特点，设计面向三维数据的高效深度神经网络，应用于物体检测、识别和分割，以及三维几何建模等计算机图形学和计算机视觉领域的支撑技术。拟解决的关键问题包括面向深度学习的三维数据表达、三维空间稀疏卷积算子设计和高分辨率高多样性三维模型合成。项目拟提出能高效处理三维数据的深度神经网络，旨在将深度学习在处理图片数据上的巨大成功推广到处理三维数据上。

卷积能够在图像上进行有效的表征学习，这是深度卷积神经网络（CNN）在图像处理领域取得巨大成功的关键因素。图像是以网格形式规则、密集且有序表示的，但点云是以点集形式不规则、稀疏且无序表示的，将适配于图像的卷积直接挪用到点云上无法进行有效的点云表征学习。为了解决这个问题，我们提出对输入点学习一种X变换，然后将其同时用于点的加权与重排列，之后再进行一般卷积。X变换的加权能编码点云的形状信息，重排列能把无序点云映射到潜在的规范顺序，经过X变换之后的卷积能够有效提取点云特征。我们提出的方法将针对图像等规则域的深度卷积神经网络（CNN）推广到能够对点云等非规则域表示的数据进行有效的特征学习，我们将其称为PointCNN。