基于多模态特征相关性挖掘的跨媒体语义子空间构建算法研究

With the rapid development of multimedia data both in type and quantity, it is more and more common that different types of multimedia data coexist, which is called cross-media data. Cross-media semantic understanding becomes one of the most hot topics recently. Traditional content-based multimedia feature analysis and semantic understanding mainly focus on multimedia data of single modality, such as: image or audio. Most of these researches are not usable for synthetical semantic understanding of multimedia data in different modalities. Multimodal features extracted from cross-media data are high-dimensional, heterogeneous and multi-rank. Considering of such feature characters, this project proposes tensor-based cross-media data representation, and then analyzes underlying multimodal correlation from both content-based features and other non-content-based features, builds low-level isomorphic subspace with correlation learning results maintained; thirdly, we use semi-supervised methods to build cross-media semantic subspace mapping model to further narrow the semantic gap. Besides, we design relevance feedback methods for cross-media knowledge refine and the mapping of new multimedia data outside original database into the cross-media semantic subspace. One to two application instances will be realized based on proposed algorithms and models to test and prove the efficience and superiority.

随着多媒体数据在类型和数量上的不断膨胀，不同类型的多媒体数据往往混合并存，表达了丰富的语义信息，称之为跨媒体数据，跨媒体语义理解也迅速成为近年来的前沿热点。传统基于内容的多媒体特征分析和语义理解研究，主要是针对单一类型的多媒体数据集，如：图像或音频。这些研究大多不能同时分析不同类型多媒体数据所表达的完整语义。从跨媒体数据中提取的多模态特征具有高维、异构、多阶的特点，针对这些特点，本项目提出以张量模型为基础建立跨媒体数据的形式化表达，进而从内容特征和非内容特征中挖掘潜在的多模态相关性，并在此基础上构建相关性保持的低维同构子空间，之后，为进一步缩小语义鸿沟，采用半监督式学习的方法建立跨媒体语义子空间映射模型，并通过相关反馈机制设计语义子空间的更新和扩展机制，实现训练集以外新数据到语义子空间的投影。最后，基于该模型设计1-2个应用实例，测试和验证本课题方法的有效性和优越性。

跨媒体数据环境中的多模态特征分析和语义挖掘广泛应用于智能信息检索、模式识别、智能机器人、大数据挖掘等领域。本项目在多模态特征潜在相关性挖掘的基础上，分析和构建跨媒体语义子空间，并实现有效的跨媒体检索，取得了一些有意义的新结果。给出了跨媒体特征的非负性学习和共生空间的有效建模算法；并且根据多模态特征中的稀疏矩阵问题，进一步提出了基于稀疏典型相关性分析的跨媒体检索方法，得到具有较强语义相关性的跨媒体稀疏表达，解决了特征分析中噪声变量的消除问题，提高了模型的稳定性。此外，还研究了未标记样本在训练过程中与标记样本的融合机制，给出了基于局部预测误差最小化的半监督学习算法。本项目的系列结果为跨媒体理论提供了新思路，在跨媒体综合语义理解与检索方面有重要的应用前景。