知识驱动的多媒体社会事件可解释关联分析

Social events association and analysis is of great significance for monitoring hot events, grasping the psychological needs of the public and meeting the major strategic needs in national security. It is also an important foundation for business operations and decision. Most of the existing researches are limited to the raw multi-modal social event data, suffering from challenge caused by non-aligned, noisy, heterogeneous, and sparse data, which bring by multi-modality, cross-platform, and cross-time-space properties of social events. In this project, we cope with challenge by incorporating event knowledge graph, focusing on the knowledge-driven social event association and trace. We try to explore how to integrate structured knowledge in deep learning, as well as the interpretability of models, which are two research hot spots in the field of artificial intelligence. The research plan has three important steps: (1) To deal with large-scale unstructured multi-modal data on the Internet, we study effective methods to build multi-modal knowledge graph and attention based structure aware multi-modal knowledge graph embedding. (2) To deal with the fragmented and non-aligned multi-modal cross-platform social event data, we study the representation algorithm using knowledge driven graph convolution based adversarial variational autoencoder. (3) To deal with the logical relation of the semantic concept and entity of event sequence, we will study interpretative association model combined logical reasoning on semantic concept and multi-modal content context, then studying events aggregation and tracing methods. We will develop a prototype platform for event association and tracing to verify the effectiveness of the proposed methods.

社会事件关联分析对于管控热点事件舆论，把握大众心理需求，满足国家安全重大战略需求具有重要意义，同时也是商业运营和决策的重要基础。现有研究大都局限使用非结构化事件数据，难以解决多模态、跨平台、跨时空性质带来的非对齐、噪声、异构、场景稀疏等挑战。本项目通过引入事件知识图谱，研究可解释关联分析应对这些挑战。着力于知识与深度学习融合、模型可解释性这两个当前人工智能领域的研究热点。主要研究工作将从三个方面展开：1）针对互联网大规模非结构化多模态数据，研究多模态事件知识图谱构建和基于注意力模型的结构感知多模态知识表示；2）针对多模态跨平台社会事件数据的碎片化和非对齐性，研究基于知识驱动的图卷积对抗变分自编码表示学习方法，可以无监督学习；3）针对事件序列高层语义实体和概念的逻辑关系，研究逻辑推理和多模态内容序列协同的可解释关联模型，建立相关事件关联聚合和溯源的方法，开发社会事件可解释关联溯源系统。

社会事件关联分析对于管控热点事件舆论，把握大众心理需求，满足国家安全重大战略需求具有重要意义，同时也是商业运营和决策的重要基础。现有研究大都局限使用非结构化事件数据，难以解决多模态、跨平台、跨时空性质带来的非对齐、噪声、异构、场景稀疏等挑战。本项目主要研究工作从三个方面展开：1）针对互联网大规模非结构化多模态数据，研究多模态事件知识图谱构建和知识表示的关键技术：无监督跨模态检索方法；2）针对多模态跨平台社会事件数据的场景稀疏和领域差异，研究预训练模型知识与深度神经网络整合学习多模态事件表示，研究基于隐语义线索提升多模态表示模型对于不同领域的泛化性；3）针对多模态跨平台社会事件数据的碎片化和非对齐性，研究基于深度学习的物联网身份认证方法，研究少量标注和含有噪声标注情况下特征提取和相似度模型，研究基于可解释时空聚类框架的事件关联聚合。对于研究内容1，提出一种基于联合模态分布的相似性无监督跨模态哈希方法；对于研究内容2-1，提出一种多层级多模态交叉注意力网络，对于研究内容2-1，提出一种多模态主题记忆网络；对于研究内容3-1，提出一种基于差分的复数卷积神经网络；对于研究内容3-2，提出一种以特征提取-重建任务为约束基于生成模型的多任务学习框架；对于研究内容3-3，提出一种基于多模态随机动态图神经网络的事件特征学习方法。为后续深入研究跨平台可解释事件关联追溯、预训模型知识表示特性和融合机制打下坚实基础。