多重相关约束下的多模态学习及其在脑图像分析中的应用

Multi-modal learning is one of the important research contents in machine learning. There are three following challenges in multi-modal brain image analysis, including: (1) severe modalities missing, (2) large diversity between the performance of strong and weak modalities, and (3) insufficient labels with high cost, which cannot be addressed by the existing multi-modal learning methods effectively. Therefore, with the application of brain image analysis, this project aims at performing the research of three scientific issues of multi-modal learning with (1) incomplete modalities, (2) strong and weak modalities, and (3) insufficient labels, respectively, by capturing the multiple correlations (i.e., inter-/intra-modality, inter-label, etc). Specifically, for incomplete modalities, this project focuses on multi-modal sample-adaptive learning and deep feature learning methods with the structural and semantic correlation between modalities. For strong and weak modalities, considering the direction and safety of strong-modal and weak-modal learning, this project conducts the study of both strong-modal guided weak-modal learning via gradual correlation and joint learning of strong and weak modalities with reliable correlation. For insufficient labels, based on both the local dependence between labels and the correlation between labels and modalities, this project focuses on multi-modal semi-supervised weak-label learning methods. Meanwhile, the aforementioned multi-modal learning methods will be applied in brain connectivity analysis, brain image segmentation and brain tumor diagnosis, respectively. In summary, this project can provide new ideas for multi-modal learning, and also is practically valuable for the application of brain image analysis.

多模态学习是机器学习的重要研究内容。在脑图像分析中，多模态脑图像常呈现“模态缺失严重、强弱模态性能差异大、标记不充足且获取困难”的特点，而现有多模态学习方法难以有效解决这些问题。为此，本项目以脑图像分析为应用背景，围绕模态不完整、模态不均衡、标记不充足情况下的多模态学习三个科学问题展开研究，通过挖掘并利用模态内外、标记间等多重相关，研究多重相关约束的多模态学习方法。针对模态不完整，运用模态间结构相关与层次语义相关，研究多模态样本自适应学习与深度特征学习方法；针对模态不均衡，考虑强弱模态互补学习的方向性与安全性，研究渐进相关的强指导弱学习和可靠相关的强弱联合学习方法；针对标记不充足，运用标记局部相关与模态标记相关，研究多模态半监督弱标记学习方法。同时，将上述方法分别用于解决脑功能连接分析、脑图像分割以及脑肿瘤辅助诊断应用。本项目可为多模态学习研究提供新的思路，且对脑图像分析应用有实用价值。

本项目聚焦机器学习中的多模态学习任务，重点研究模态不完整、模态不均衡、标记不充足情况下的多模态学习三个科学问题，并且在不同应用领域验证了多模态学习算法的有效性。具体地，针对模态不完整情况下的多模态学习问题，本项目创造性地提出了一系列多模态算法，在心脏图像分割与跨模态人脸自动生成应用中得到有效验证，利用基于近邻生成对抗网络解决半配对图像转化问题。相关成果发表在MICCAI、ACCV、IJCNN等国内外学术会议上。针对模态不均衡情况下的多模态学习问题，本项目深入挖掘模态间的渐进相关性，同时利用样本自适应解决大脑肿瘤分割问题。针对源域和目标域图像的分布不同设计了基于条件生成对抗网络的无监督域适应图像分类方法。针对多标签跨模态检索问题，提出一系列哈希学习方法。针对小样本无监督域适应问题，提出一种图像到类别的稀疏相似性编码方法和基于元学习的小样本无监督域适应方法。相关成果发表在 IEEE JBHI、IEEE TMI、Pattern Recognition、AAAI、ACM MM、ICME等国内外学术期刊与会议上。针对标记不充足（或含噪声）情况下的多标记学习问题，本项目综合运用稀疏学习、低秩表示、标记子空间学习以及字典学习等方法挖掘标记相关性，创造性地提出了一系列多标记分类学习方法。相关成果发表在NPL、Applied Intelligence、KBS、PAKDD、ICME等国内外学术期刊与会议上。针对空-谱联合的特征学习以及数据噪声问题，本项目在 Geoinformatica、IEEE TCSVT、Pattern Recognition 等国内外学术期刊上发表了一系列工作。本项目聚焦单视角三维人脸重建任务，开展了部分前期工作，为后续多模态三维人脸重构奠定基础。相关成果发表在ICIP国外学术会议上。本项目可为多模态学习研究提供新的思路，且对实际多模态图像分析应用有较好的实用价值。