面向图像盲超分辨的模块化深度网络建模与学习方法研究

Recent years have witnessed tremendous success of deep convolutional neural networks (CNN) for image super-resolution. In general, the existing CNN-based super-resolution methods 1) assume the down-sampling model is known in advance; 2) and adopt peak signal to noise ratio (PSNR) or structural similarity (SSIM) index as objective functions. Methods with such a learning strategy, however, have typical drawbacks in practical applications. On the one hand, the real world down-sampling (RWDS) model is usually unknown, and consequently, annoying noises and compression artifacts would be inevitably introduced when the RWDS model does not match the assumed one. On the other hand, objective functions based on PSNR or SSIM tend to over-smooth image textures. Hence, it is an urgent problem to implement the blind SR in current research. To address the above issues, the project will focus on the following four studies: (1) investigate the architecture design, strategies of cascading and joint learning for image reinforcement and super-resolution networks; (2) design an effective objective function which can preserve both high semantic information and visual qualities; (3) propose alternative solutions and learning algorithms which can be trained without ground-truth high-resolution images, or more precisely, the simulation strategy and generative adversarial learning strategy will be studied; (4) extend single image super-resolution to video super-resolution. To sum up, this project aims to develop more practical image super-resolution methods for real world applications. All these studies will provide useful tools and insights for other image enhancement methods. In previous work, we have developed a series of image denoising and enhancing methods which have validated the effectiveness and feasibility of the proposed project. We strongly believe that the project will lead to valuable and impactful research outputs.

目前，基于深度卷积网络的超分辨算法需基于下采样模型已知的仿真数据进行训练，目标函数则多基于PSNR或SSIM等指标构造。然而，实际应用中下采样模型大多未知，并伴随噪声或压缩伪影，且无对应的真值高分辨率（SR）图像。此外，采用PSNR或SSIM作为优化目标也常导致过度光滑而损失图像细节。如何实现图像的高品质盲超分辨成为该领域学术研究中亟待突破的关键问题之一。为此，本项目拟研究：1)增强网络与超分辨网络的设计、级联方案及联合学习算法；2)以保持高层语义和提升视觉效果为目标的网络结构和目标函数设计；3)无真值SR图像情况下的替代解决方案和学习算法，包括仿真策略和生成式对抗学习策略；4)从图像推广至视频超分辨。项目有望发展出更符合实际应用需求的超分辨模型，对其它图像增强方法的发展也有一定的借鉴和指导意义，具有重要的学术价值。项目组在图像去噪与增强等方面具有较好的研究基础，可为项目顺利实施提供保障。

目前，基于深度卷积网络的超分辨算法需基于下采样模型已知的仿真数据进行训练，目标函数则多基于PSNR或SSIM等指标构造。然而在实际应用中下采样模型往往是未知的，并伴随噪声、模糊或压缩伪影等退化，且没有与退化图像所对应的真值高分辨率（SR）图像。此外，采用PSNR或SSIM作为优化目标也常导致过度光滑而损失图像细节。如何实现图像的高品质盲超分辨成为该领域学术研究中亟待突破的关键问题之一。. 为此，本项目提出并实现了基于小波变换的图像复原基础构架，构建起了基于级联卷积网络的盲真实图像超分辨框架，发展了基于多引导图和自适应特征融合的真实退化人脸图像复原方法，完成了基于不确定性深度网络的真实图像去噪集成框架等计划研究内容。其中，项目所提出的MWCNN方法的谷歌学术引用累计500余次（截止2022年底），已经成为图像复原领域较为经典的文献之一。通过项目研究，发表国内外学术论文20余篇（SCI/EI 收录20余篇），包括TIP、TMM、TCSVT等IEEE汇刊论文5篇，CVPR/NIPS/ECCV等CCF A类会议论文5篇。获得黑龙江省自然科学类一等奖，发明专利1项；培养9名博士和5 名硕士，为国家输送模式识别、计算机视觉等方面的优秀人才；派遣博硕士生2人次赴香港理工大学交流访，线上参加3次相关领域顶级或重要国际会议。. 本项目所实现的以MWCNN、CBSR、ASFFNet、BDE为代表的一系列深度视觉计算模型与方法在性能上达到同期最优，具有重要的学术价值。此外，这些方法也逐渐形成了一套面向实际应用需求的图像复原与增强方法体系雏形，对相关方法的落地应用具有指导意义。