低质多模态数据深度融合算法研究

In the big data era, a wide array of multi-source and heterogeneous data have been generated in multiple modalities. It is significant to explore the huge value hidden in the mass through the complementary knowledge learning among multimodal data. However, the low-quality characteristic of multimodal data, such as inaccuracy, incompleteness and unbalance, poses great challenges to the design of data fusion methods. Deep neural networks (DNN) can locate the deep feature abstraction of data through multi-layer nonlinear transformation and thus promote the semantic fusion. Inspired by this, the project will extend DNN to study the corresponding deep fusion algorithms for low-quality multimodal data, including: (1) A new deep feature correlating model for inaccurate multimodal data will be constructed. Through noise reduction and feature abstraction of DNN, and coupled optimization of modality specific and shared features, the accuracy of the fusion representation across modalities can be improved. (2) A novel deep feature mapping model for incomplete multimodal data will be designed. By exploiting the high-level feature integration characteristic of DNN, the deep mapping networks and common features across modalities can be jointly optimized. Thus, it will boost the data fusion results. (3) A new deep feature transfer model for unbalanced multimodal data will be developed. Through the multilayer correlating and matching for features across modalities, the semantic bias between them can be effectively bridged, which will promote the correlation of transfer features for data fusion. Based on the above research and innovative design, the low-quality multimodal data can be deeply fused and shared, and promising features can be located and extracted, which will boost the tasks of data classification and prediction.

大数据时代多源、异构等多模态数据广泛存在，通过模态间互补学习挖掘数据中隐藏的价值具有重要意义。然而多模态数据的不精确、不完整和不均衡等低质特性为融合方法的设计提出了巨大挑战。深度神经网络能够通过多层非线性转换，学习数据的深层抽象特征，促进相似语义融合。因此，本项目扩展深度神经网络到低质多模态数据，研究以下三个问题：（1）不精确多模态数据深度相关融合，通过深度神经网络的降噪和特征抽象，及模态私有特征和共享特征的耦合优化，提高融合表示的精度；（2）不完整多模态数据深度匹配融合，利用深度网络的深层特征整合特性，联合优化模态深度匹配网络和共享特征，提取更加准确的融合结果；（3）不均衡多模态数据深度迁移融合，通过跨模态特征多层相关匹配，有效弥补不均衡模态间的语义偏差，提升迁移融合特征的相关性。实现低质多模态数据的深度融合与特征提取，提高数据分类与预测的准确性。

大数据时代多源、异构等多模态数据广泛存在，通过模态间互补学习挖掘数据中隐藏的价值具有重要意义。然而多模态数据的不精确、不完整和不均衡等低质特性为融合方法的设计提出了巨大挑战。深度神经网络能够通过多层非线性转换，学习数据的深层抽象特征，促进相似语义融合。基于此，耦合深度神经网络和多模态数据的上述特点，本项目研究以下三个问题：（1）不精确多模态数据深度相关融合，通过深度神经网络的降噪和特征抽象，及模态私有特征和共享特征的耦合优化，提高融合表示的精度；（2）不完整多模态数据深度匹配融合，利用深度网络的深层特征整合特性，联合优化模态深度匹配网络和共享特征，提取更加准确的融合结果；（3）不均衡多模态数据深度迁移融合，通过跨模态特征多层相关匹配，有效弥补不均衡模态间的语义偏差，提升迁移融合特征的相关性。实现低质多模态数据的深度融合与特征提取，提高数据分类与预测的准确性。.项目执行过程中，针对不精确、不完整和不均衡等低质多模态融合分析问题，开展了深入研究，提出实现了一种非负相关和不相关的特征协同学习方法、一种基于加权稀疏低秩表示的不完全多模态数据融合方法、一种双对齐自监督不完整多视图子空间聚类网络、非对称多模态变分自动编码器和一种新的跨模态融合图像恢复方法等。并根据团队实际项目需求，开展了时序多模态融合和多模态医学应用等研究，提出实现了基于非负矩阵分解的增量多视图相关特征学习、基于多神经网络的多模态集成学习、多模态医学影像分割等方法。发表学术论文20篇，申请和授权国家发明专利5项，获得国际会议最佳论文奖1项，获得海南省科学技术进步一等奖1项，培养毕业研究生5名。获取整理了多模态工业数据集1个、多模态医学数据集4个。为今后多模态融合理论研究和多模态理论在医学、工业等领域应用研究及关键技术突破奠定了深厚的基础，能够服务于多模态融合相关的人工智能与领域交叉产业，产生一定的经济价值和社会效益。