面向三维展示的数据驱动多视角三维重建方法研究

Nowadays, the form of online exhibition for commodities are mainly 2D images, which has major drawbacks, such as flat visual effect and lack of user interaction. Introducing 3D information into the commodities exhibition will largely enhance the user experience and boost market promotion, and thus have huge application prospect. Multi-view stereo reconstruction methods can easily and conveniently use off the shelf cameras, such as mobile phones, digital cameras and webcams, to reconstruct 3D shape of the interested objects according to multiple images from different views. However, most of man-made commodities, such as tables, chairs, teacups and household electrical appliances, have high proportion of low-textured surface due to their material and design style. Using state-of-the-art multi-view stereo methods for reconstruction of commodities can lead to low-quality results with low completeness which are not suitable for 3D exhibition...Based on this background, in this proposal, we plan to use data-driven methodology to explore robust multi-view stereo methods for reconstruction of low-textured man-made objects. Our research pipeline in this proposal is beginning with data collection, then including multi-view dense depth estimation, 3D model generation and detail enhancement of small 3D geometric features, and ending with 3D reconstruction prototype system design and evaluation. According to this research pipeline, we will focus on several important problems, which cover data collection and database design, dense depth maps estimation based on confidence measure and depth maps hole-filling, 3D model reconstruction based on analysis sparse representation and structural semantic prior, adaptive fusion of stereo and shading cue base on reprojection error minimization framework, and finally, we will build a 3D reconstruction prototype system of low-textured commodities and evaluate its capability on 3D reconstruction and 3D exhibition.

当前商品在线展示形式基本是二维图像，存在扁平化、单一化、缺乏交互等问题，引入三维信息将可增强用户体验和商品的推广效果，具有巨大的应用前景。多视角三维重建方法可方便快捷地利用现有摄像设备，如手机、数码相机等拍摄物体不同角度图像重建其三维形状，但多数人造物品如桌、椅、茶杯、电器等由于材质和设计风格等因素包含大面积的匀质弱纹理区域，采用现有方法将导致重建结果完整度极低，展示效果无法满足需求。.以此为背景，本项目基于数据驱动方法研究对弱纹理鲁棒的多视角立体重建算法，以数据准备－多视角稠密深度图生成－三维模型生成－三维几何细节提升－三维重建系统原型设计与评估为主线，重点研究训练数据的采集和数据集建立、基于置信度估计与深度图修补的稠密深度图生成、基于分析式稀疏表达与结构化语义先验的三维模型生成，重投影误差框架下立体与明暗信息自适应融合，最终研究面向三维展示的弱纹理商品三维重建原型系统设计与评估。

多视角立体重建通过捕获多角度交叠的图像数据重建场景表面，具有低成本和灵活性的优势。然而人造场景广泛存在弱纹理等挑战性表面外观，导致多视角立体重建方法的重建完整度较低，限制了多视角立体重建方法在商品三维展示和虚拟现实等对三维重建完整度要求较高的领域的应用。本项目在资助期间对弱纹理场景的三维重建进行了深入研究，主要研究内容包括：面向大尺度数据和置信度驱动的多视角三维重建算法；面向小运动视频的人造场景的鲁棒三维重建方法；面向多视角立体的深度图置信度预测算法；面向室外场景自由视点合成的深度图补全；基于多尺度金字塔注意力机制的单目深度估计；以及面向文物和古迹三维重建的大尺度三维重建算法应用等。本项目从置信度预测，基于PatchMatch的大尺度多视角立体，深度图补全，单目深度图和多视角立体深度图的融合，人造场景的视点合成和窄基线三维重建等核心子问题展开研究，显著提升了方法在人造场景下的三维重建的性能。特别地，在公共数据集上的大量评估表明，提出的算法可以有效的复原室内外人造场景和物体的弱纹理区域的三维表面，显著提升重建完整度的同时保留表面的几何结构，方法可以处理高清图像输入并具有较好的泛化能力，代表性工作包括：算法在ETH3D 高清图像评测数据集上连续7个月排名第一，并发表在图像处理顶级期刊IEEE Transactions on Image processing. 三维重建方法还应用在文物和古迹的三维重建和展示，获得虚拟现实大会最佳论文提名奖。..弱纹理表面三维重建是多视角立体重建的核心难点问题，本项目以数据为驱动，提出初始深度估计-置信度估计-深度图插值-深度图精细化的技术路线。 核心创新包括数据驱动的置信度估计，单目深度估计，单视角和多视角深度图的融合，面向高清图像输入的三维重建等，为后续研究奠定了基础并提供了有价值的研究思路。本项目显著提升了当前基于图像的三维重建算法在人造室内和室外场景下的三维重建的完整度，研究成果预期可以应用在商品三维展示，虚拟现实，遥感，机器人导航等领域。