基于多模态融合机制的视频语义表征方法研究

With the development of Internet applications and storage technology, video data has increased explosively. The video data has many distinct features, such as information diversity, high redundancy, low degree of abstraction and unstructured characteristic. On the one hand, several types of media data including image, audio as well as text are contained in videos, and there are temporal correlations among them; on the other hand, video data has a strong logic in terms of content. These features bring great challenges to intelligent analysis of video data. Based on the multimedia analysis technology, this project comprehensively applies a series of technologies from several popular subjects, involving deep learning, image processing, pattern recognition and theme model, to study video data from the following three aspects: (1) construct an adaptive deep model by analyzing video visual feature, and create a video representation from the visual view. (2) study the effective fusion mechanism based on multimodal information within video data, and explore some invariant relationship from the multivariate heterogeneous data. (3) build a video semantic representation model with domain knowledge, which can adapt to different application scenarios for different types of video data. The accomplishment of this project will not only enrich the application of the theme model, but also provide new insights into the intelligent analysis of video data.

随着互联网应用及存储技术的发展，视频数据呈爆炸式增长。视频具有信息量大、冗余度高、抽象程度底和非结构化等特性：一方面，视频中存在图像、音频和文本等多种媒介数据，且相互之间呈现时序关联性；另一方面，视频从内容上又具有很强的逻辑性。这些特性给视频数据的智能分析研究带来巨大挑战。本项目以多媒体分析技术为基础，以视频为研究对象，综合运用深度学习、图像处理、模式识别和主题模型等技术，拟从以下三个方面展开研究：(1)研究视频视觉特征的自动学习和提取机制，构建自适应特性的深度模型，从视觉层次建立对视频的描述；（2）研究视频数据中多模态信息的有效融合机制，以期在多元异构数据中寻找某些不变关系；（3）研究结合领域知识的视频语义表征，根据视频类别特点和不同的应用场景，在多模态融合基础上构建一套相应的语义表征模型。本项目的完成不仅能够丰富主题模型的应用领域，还将对各领域视频的智能分析研究提供新的视角和理论突破。

依托本项目，课题组主要围绕以广告视频为代表的视频表征和Logo目标检测进行研究。一方面，在进行视频表征构建过程中，不仅考虑了视频中的视觉、音频等特征，还融合了视频的领域特性。更进一步，构建了一种包含高层语义特性的视频表征算子。另一方面，课题组对以广告视频为代表的短视频进行进一步研究，具体来说，对广告视频中的Logo品牌信息进行了挖掘。目前阶段，实现了大规模Logo数据集的构建和图像中Logo目标检测。.课题组在研究过程中发表了多篇学术论文，其中SCI论文6篇，CCF A类1篇，申请发明专利3项目，其中授权1项。协助培养硕士研究生5名。