多路聚合的跨媒体模式分析与知识推理研究

This project proposes multi-way aggregation (MAG) method which aims at remedying the heterogeneous representations of different media data, the semantic gap between the underlying representations and high-level semantics in each media, and the measurement and optimization of cross-media complex data model. Based on the multi-information and conditional mutual information, we model the relationship of the latent variables in the MAG method, which builds a basic model for cross-media data pattern analysis and knowledge reasoning. By making use of the affinity propagation method, we will construct a novel cross-media information propagation mechanism to remedy the heterogeneous representations of cross-media data. To deal with the semantic gap in each media data, we will construct a visual knowledge graph and design a cross-media reasoning mechanism. For the measurement and optimization of cross-media complex data model, we will build a synthesis compression function of the MAG method, which can guarantee the robustness of the deep measurement function. The research of the information propagation, knowledge reasoning and the measurement and optimization methods of cross-media data models are original and have potential theoretical and practical value.

本项目提出多路聚合方法，拟解决跨媒体数据中不同媒体间的异构问题，各媒体内部底层表征与高层语义间的语义鸿沟问题和跨媒体复杂数据的度量和优化问题。基于多信息和条件互信息，对多路聚合方法相关模型中变量间的关联关系建模，为跨媒体数据模式分析和知识推理构建基本模型；以近邻传播方法为核心，构建多路聚合方法中的跨媒体信息传播机制，解决不同媒体数据的异构问题；构建面向跨媒体数据的视觉知识图谱并设计跨媒体知识推理机制，解决跨媒体数据各媒体内数据的语义鸿沟问题；基于多路聚合方法的综合压缩函数，构建跨媒体复杂数据模型的度量和优化策略，提高深度度量方法的鲁棒性。本项目在跨媒体信息传播、知识推理和复杂数据模型的度量优化等方面的研究是原创性的，并且具有重要的理论和应用价值。

针对跨媒体数据模式分析中不同媒体数据的异构问题、各媒体内底层数据表征与高层语义间的语义鸿沟问题和跨媒体复杂数据模型的度量优化问题，本项目对多路聚合模型及算法、跨媒体信息融合与共享机制、跨媒体知识推理机制和跨媒体深度模型度量和优化展开深入研究，并取得了一些阶段性研究成果，包括多任务协作模型、跨媒体信息融合模型、视觉上下文模型、联合聚类分析算法、多视角信息融合与共享算法、基于知识图谱的跨媒体推理模型、深度模型的度量和优化方法。相关成果发表在中国计算机学会推荐A类国际会议AAAI、中国计算机学会推荐B类国际会议SIAM SDM、中科院一区期刊IEEE Transactions on Industrial Informatics、Information Fusion、Expert Systems with Applications、Information Sciences。另外，相关阶段性成果已向中科院一区期刊IEEE TNNLS（二审）、IEEE TCYB（三审）投稿，并取得了初步的正面评价。