基于场景本征属性感知的图像重渲染研究

Image re-rendering refers to transforming a given image or set of images so that the generated image differs from the original one in perspective, lighting, color, texture, or shape. Image re-rendering has wide applications in augmented reality, image recognition, and film and television production. The traditional scheme for image rerendering is to separately recover the intrinsic scene characteristics (surface shape, surface reflectance, scene lighting, and shading), then edit part of the intrinsic properties according to particular rerendering tasks and finally re-synthesize a new image. However, the recovery of intrinsic scene characteristics from images is an ill-posed inverse problem, which brings great limitations to the traditional image re-rendering approaches. The project intends to employ the advanced machine learning methods such as deep learning and manifold embedding to investigate the model of perception and control of intrinsic scene properties from images, and further study new re-rendering methods for general images (such as the images with non-Lambertian surface and non-point light source). The proposed scheme does not require explicit recovery for intrinsic scene components from images. The research program of project focuses on a unified modeling for two-way tasks, i.e., the perception of intrinsic scene characteristics and image re-rendering, which makes the research program more systematic. A smooth implementation of the project will develop the image processing theory and technology, while improving the overall level of research group.

图像重渲染是指对给定的图像或图像集进行变换，使新生成的图像内容较原图像具有不同的视角、光照、颜色、纹理、形状等特征。它在增强现实、图像识别、影视制作等方面具有广泛应用。实现图像重渲染的传统思路是对图像中的场景本征成分（表面形状、表面反射率、场景光照、阴影）进行分离重建，然后根据渲染任务对部分本征属性进行编辑并重新合成新的图像。然而，根据图像重建场景本征成分是一个病态反问题，这给传统的图像重渲染方法带来极大局限。本项目拟以深度学习、流形嵌入等先进的机器学习方法为工具，研究对图像本征属性进行有效感知与精准操控的模型，并在此模型的基础上研究面向一般图像（如非朗伯表面、非点光源环境）的重渲染方法，避免显式地对图像本征成分进行重建。项目研究方案侧重对场景本征属性感知与图像重渲染双向任务的统一建模，具有系统性。项目的顺利开展将推动图像处理相关理论与技术的进一步发展，同时提高团队整体科研水平。

项目主要研究视频图像本征属性（场景表面形状、表面反射率、光照、运动）感知理论与方法，并在此基础上研究面向一般图像的重渲染方法，即对给定的图像进行变换，使新生成的图像内容较原图像具有不同的视角、光照、颜色、纹理、形状等特征。它在增强现实、图像识别、影视制作等方面具有重要应用。项目在如下方面取得了进展：1）在图像本征属性感知方面，构建了一个人工合成的本征图像数据库，突破图像本征成分数据库缺乏问题；提出基于深度学习的图像本征属性重建模型，既支持RGB图像也支持RGB-D图像作为输入；提出基于偏微分学习的图像本征属性估计方法，模型的学习仅需要少量训练样本（一般仅需要十几个样本）；提出基于低秩稀疏分解模型的图像本征属性重建方法；提出一种基于神经网络的图像光照估计方法，用渲染数据训练后的模型可以对单张图像估计相应场景的光源方向、光源距离和光源能量。2）在图像重渲染方面，研发出基于三维重构的人脸自动化妆系统，进一步扩展到视频应用；提出一种基于生成式对抗网络进行图像去模糊的方法；提出基于对抗生成网络的人脸重加光深度学习模型；提出基于对抗生成网络的人脸光照迁移深度学习模型，可以适应任意光照类型之间的迁移。整体上，项目成果在国际权威学术期刊和会议发表论文30篇，其中包括IEEE汇刊论文9篇、中国计算机学会（CCF）推荐的顶级会议4篇。申请国家发明专利8项，其中2项获得授权。