融合多深度的复杂场景多视点采样与重建的基础理论与关键技术

Most of the existing multi-view video acquisition systems are based on multiple texture cameras and lead to two problems: the huge data volume, which comes from multiple viewpoints and bring challenges to storage and transmission, and the lack of depth information, which leads to difficulties in 3D video and free-viewpoint video generation...Considering the limitations above, this project proposes a novel framework based on multiple depth cameras and less texture cameras for multi-view acquisition and reconstruction, and in the framework we consider multiple technique problems as follows. ..Firstly, we break the limitation of traditional sampling theories, put forward spectral analysis method in multiple depth camera sampling, describe the complex scene with higher accuracy, and develop novel sampling theories. Secondly, we research the interference among depth cameras and propose innovative methods for interference cancelling and precision enhancement based on orthogonal signals，MIMO signal processing, and plane-sweeping. Thirdly, we develop rate-distortion models for depth coding and propose multiple depth coding methods. Finally, we propose novel scene reconstruction and view synthesis algorithms based on user perception and viewpoint prediction. The algorithms take full advantages of depth information, solve the problems in view synthesis, and provide users with more realistic experiences...With the multiple-depth-based acquisition and reconstruction system, the project benefits our country's innovation research in scientific fields, which includes multi-view video acquisition, sampling theory, interference canceling, joint rate-distortion theory, and also provides technical support in developing depth acquisition devices with our independent intellectual properties.

现有的多视点视频采集系统都是以多纹理为基础，其数据量巨大,不便于网络传输；深度信息不足,不便于立体视和自由视的生成。本项目针对这一局限，提出全新的以多深度少纹理为基础的多视点采集和重建架构，研究该框架下复杂场景采样与重建的科学问题：突破传统全光采样理论，提出基于深度采样的频谱分析方法，完善复杂场景的数学描述，发展多视点采样新理论；揭示深度干扰机理，创新性地提出基于信号正交化特征和MIMO信号检测机制以及平面扫描技术的深度图干扰消除方法，重构精细化深度图；构造深度率失真模型，建立多深度编码机制；提出用户感知和视点预测机制，充分融合多深度场景信息，解决场景绘制中的难题，创造更具真实感的自然体验。形成一套融合多深度的复杂场景多视点采样与重建研究体系，为我国在多视点采集、采样理论、干扰消除、联合率失真等科学领域的原始创新，为发展自主知识产权的深度成像、自由视电视等领域奠定重要的理论基础和技术支撑。

交互游戏、AR/VR，全景直播等应用的流行，表明有效体现人类自然感官体验的多视点视频技术是未来发展的重要领域。本项目旨在解决融合深度的复杂场景多视点采集和重建架构中的科学问题，围绕采样理论、多深度干扰、多视点多深度编码、复杂场景重建等系列难点展开研究。项目从遮挡、非朗伯反射等复杂场景出发，分析其频谱结构，推导出精确的采样率，提出融合深度信息的采样方法，构造深度信息描述的数学模型，并应用于复杂场景采样。项目还建立基于光线距离、二维平面及三维空间的覆盖域模型优化相机的位置和方向，形成了一套完整的多视点采样理论体系。多路深度图获取方面，构建了深度相机干扰模型，提出空域正交、平面扫描、正交光强调制等方案消除深度相机之间的干扰，设计多频率混合结构光、区域自适应、空洞平面假设、联合时空约束等方案提升深度图质量，并提出了基于纹理与深度的边缘匹配度量的无参考的质量评价方法，为高质量多视点深度图的获取提供了技术支撑和理论依据。多视点多深度编码方面，研究了多终端信源编码的边界理论问题，为多视点编码提供理论依据；分析深度失真影响，构建深度-虚拟视点联合率失真模型，提出基于模型的联合码率分配、边缘敏感的深度编码以及基于深度线索的联合编码等方法；设计了视空互补的迭代优化、直方图重构校正等优化方案，提升了编码质量。场景重建方面，建立图结构稀疏正则化模型描述场景结构，并利用物体结构相似性重构三维场景；提出多深度线索融合，光流校正和图割模型等方法融合深度，设计光栅扫描、突变点消除等算法提升视点绘制质量。项目期间，研究单位搭建了一系列实验平台如：可编程结构光深度采集平台和多路同步采集平台，为深度干扰消除的研究奠定了硬件基础；多光照光场成像采集系统实现了多视点视频采集；交互式3D传输与显示系统实现了2D-3D视频转换，视点自由切换。本项目成果为自由视点电视、全景直播、虚拟现实等前沿应用奠定重要的理论基础和技术支撑。