与主观感知一致的立体图像和视频视觉质量客观评价与增强研究

The rapid development of stereoscopic 3D (S3D) devices and the unique depth perception of S3D image and video bring a rich demand for high quality S3D image and video. But S3D market is suffering from the shortage of high quality S3D image and video and the damage to human visual health due to watching low quality S3D image and video. So we aim to enrich S3D image and video and monitor the quality of S3D image and video by automaticly assessing and enhancing the visual quality of amateur S3D works. The research includes S3D image saliency detection based on sparse matching features, the assessment and enhancement of color consistency between left and right views of S3D image and video, and the assessment and enhancement of depth information visual comfort of S3D image and video. The utilization of sparse feature matching instead of time-consuming dense stereo matching can greatly improve the time performance of saliency detection. An optimization method is employed to make up the incomplete of depth information due to sparse feature matching. Subjective perception consistent visual quality assessment and enhancement methods will be proposed by combining user study and machine learning and using visual saliency. The calculation of target visual comfort disparity will be formulated as a quadratic minimization problem to avoid time-consuming 3D reconstruction. A content-aware image warping algorithm will be presented to achieve the balance among visual comfort enhancing, distortion avoiding/shape preserving, spatio-temporal coherence preserving, and content maintaining. Finally, a S3D image and video authoring and visual quality monitoring system based on iterative assessment and enhancement will be developed. The system can both enrich S3D image and video and provide effective techniques for maintaining a visual health S3D digital media environment.

本项目针对三维立体市场存在的高质量三维数字媒体片源匮乏以及低质量三维媒体对人类视觉健康的损害等问题，提出自动评价和增强业余三维立体作品视觉质量、丰富三维片源并监控三维媒体视觉质量的解决方案。研究内容包括基于稀疏匹配特征的立体图像显著性检测、立体图像和视频左右视图颜色一致性评价与增强以及深度信息视觉舒适度评价与增强方法。借助稀疏特征匹配的高效性提高显著性检测算法的时间性能，采用优化技术弥补匹配特征稀疏性导致的深度图不完全；结合用户调查与机器学习方法、以视觉注意为指导，建立与主观感知一致的视觉质量客观评价；避免耗时的三维重建步骤，将视觉舒适的目标视差求解表示成二次最小化问题；采用图像变形技术，实现增强视觉舒适度、避免扭曲、保持时空连续性和保留内容间的平衡。实现基于迭代式评价和增强策略的立体图像和视频创作与视觉质量监控系统，丰富三维片源，并为建立视觉健康的三维媒体环境提供有效的技术支持。

为丰富三维图像和视频片源，本项目对与主观感知一致的立体图像和视频视觉质量客观评价与增强方法进行了一系列研究。本项目主要工作包括：提出基于稀疏匹配特征的立体图像显著性检测方法、建立立体图像颜色一致性评价数据集、提出立体图像颜色一致性评价和增强方法、提出时空一致的立体视频颜色一致性增强方法及3D图像深度信息视觉舒适度评价和增强方法。在完成研究计划的基础上，采用深度学习技术深入研究本项目的若干问题及相关问题，包括基于边缘感知和跨模态特征采样的RGBD图像显著性检测方法、基于循环神经网络和多层注意力的专业立体视频舒适度分类方法、基于立体校正和视差重映射联合优化的立体图像舒适度增强方法、基于自注意力机制的立体图像视觉舒适度增强方法、基于多任务学习和匹配失真表示的无参考立体图像质量评价方法等。上述研究成果有助于丰富三维片源，并为建立视觉健康的三维媒体环境提供有效的技术支持。