面向三维视频编码的智能视点合成预测方法

Multiview video plus depth is an important representation format of 3D video and widely studied by researchers. Due to the presence of videos from multiple views, the amount of data significantly increases comparing with the conventional 2D video. Therefore, efficiently reducing the redundancy among multiple views is one of the key technologies to achieve efficient 3D video coding. View synthesis prediction, based on the depth-image-based rendering (DIBR) technique, enables the prediction of the to-be-coded view from the reference view by 3D warping. However, in the 3D warping process, the pixels in the reference view cannot be directly mapped to integer pixel locations, and thus interpolation to integer pixels is needed, leading to geometric distortion. Thus the performance of view synthesis prediction is degraded. This project focuses on the view synthesis prediction for 3D video coding. The relationship between the warping location and the pixel quality in the warped image will be investigated, and a new method on geometric distortion rectification and adaptive pixel-level view blending will be developed by taking advantage of the deep learning technologies. The new method will enhance the quality of view synthesis, and thus improve the efficiency of the view synthesis prediction and 3D video coding. The research of this project is of great scientific importance for the further development of 3D video coding.

多视点加深度视频是一种重要的三维视频表示格式并受到学者的广泛关注，由于多个视点的出现，三维视频数据量相对传统二维视频急剧增加，因此，如何有效去除视点间相关性是实现高效三维视频编码的关键技术。视点合成预测利用基于深度的视点合成技术将已编码的参考视点映射至待编码视点作为参考进行预测，然而，在视点合成的三维映射过程中，由于参考视点像素不能准确映射至待编码视点的整数像素位置，导致映射图像中存在几何失真，因而视点合成预测效率较低。本项目围绕三维视频编码中的视点合成预测展开研究，分析三维映射几何位置与像素质量的关系，通过构建深度学习神经网络，充分利用映射几何位置信息建立三维映射几何失真校正和像素级自适应视点融合模型，突破传统视点合成无法进行自适应校正及增强的技术瓶颈，实现智能视点合成预测并提升其在三维视频编码中的效率，对推动三维视频编码的发展具有重要的科学意义。

随着三维视频的逐渐普及，三维视频相比二维视频的更大数据量及更高体验感要求对三维视频的高质量视点合成和高效编码提出了更大的需求。本项目针对三维视频视点合成中几何失真问题和编码视频质量增强开展了相关研究：分析像素三维映射几何位置信息对映射图像质量的影响，利用可变形卷积，提出了基于几何映射误差的视点增强，实现三维映射中的几何失真校正；同时研究像素几何位置信息对视点融合的影响，建立基于像素几何位置信息的自适应像素级视点融合；对于融合图像，进一步使用部分卷积进行空洞填补，提升最终视点生成质量。在编码视频增强方面，分析编码视频的失真来源，提出基于全局图像信息和基于局部块失真信息的编码视频质量增强方案；提出了基于多尺度偏移计算的多帧视频编码增强方法，改善可变形卷积梯度回传不稳定问题，显著提升了编码视频质量，达到25.6%的相对码率节省。成果方面，发表论文10篇，包括国际期刊论文4篇和会议论文6篇，包括IEEE TBC、ACM MM等；提交专利申请7项，其中3项已授权。