多模态流形协同的跨媒体计算及其社交多媒体应用研究

Social multimedia intelligence acts a key role in the development of national technology and economy. As cross-media cognition is put forward, cross-media computing becomes a hot research topic. However, both multimedia “semantic gap” and user “intent gap” from information overload are still obstacles undesirable in social multimedia applications. The study focuses on investigating cross-media computing problem and proposes multimodal manifolds based collaborative association estimation to reveal social multimedia correlations. With analyzing consistency and complementary attributes of multi-modal information, the project would construct multi-modal manifolds based collaborative computing model to perform cross-media association estimation and bridge both the multimedia semantic gap and user intent gap. Specifically, this project employs manifold structure of multi-modal information to propagate cross-modal association along the manifold distribution. Furthermore, a scalable manifold-collaborative computing model is formed over local manifold to propagate cross-modal associations progressively over multi-modal manifolds. It aims to estimate global cross-modal associations and bridge social multimedia sematic gap and user intent gap. The topic is of great importance in illustrating collaborative learning mechanism of multi-modal manifolds and revealing rules of cross-media collaborative cognition in social multimedia applications.

社交多媒体智能化是关系到国家科技进步和经济发展的关键问题，随着跨媒体认知的提出，跨媒体计算成为前沿热点。其中，多媒体信息的“语义鸿沟”以及信息过载带来的用户“意图鸿沟”一直是社交多媒体智能化的瓶颈问题。本项目针对跨媒体计算问题，以社交多媒体信息为研究对象，提出多模态流形协同学习的方法，研究跨媒体信息关联，分析多模态信息的一致性与互补性，建立多模态流形关系协同学习的跨媒体计算模型，实现跨模态信息相关度估计，贯通多媒体语义鸿沟与用户意图鸿沟，推进社交多媒体智能化的发展。具体的，本项目拟采用多模态信息的拓扑流形关系作为跨模态信息关联的传播途径，同时考虑社交多媒体信息的动态变化特性，建立具有流形扩展能力的多模态协同计算模型，通过多模态流形协同的相关度传播，估计全局信息的跨模态相关度，建立社交多媒体语义关联。项目研究成果对阐明多模态流形协同学习机制、揭示社交多媒体中的信息协同认知规律具有重要意义。

针对社交多媒体环境的语义鸿沟以及用户意图鸿沟问题，本项目以社交多媒体信息服务为研究对象，主要研究用户兴趣的跨模态协同机制，利用用户交互的异构图流形关联传播用户兴趣，建立多模态协同的社交多媒体计算模型，同时研究用户兴趣的局部自适应扩展计算，以适应社交多媒体环境动态多样化的服务特点。基于上述研究工作，提出了面向用户兴趣的个性化跨模态协同交互推荐方法，揭示了用户决策的学习模式；实现了基于流形传播策略的用户兴趣相关度匹配算法，构建了异构图兴趣传播的学习途径；完成了多模态协同的图映射交互模型，探索了社交多媒体环境下多模态协同促进的学习趋势；建立了图流形局部自适应计算的扩展模型，实现了用户兴趣导向的动态匹配。基于多模态异构图,项目研究成果揭示了用户兴趣、用户关联、用户决策等多个层面的发展趋势，开拓了社交媒体信息服务新格局，有效促进了社交多媒体环境智能化发展。