社会媒体中图像隐含语义理解方法研究

In social media, the goal of image implicit semantics understanding is to analyze the views, intents, etc. that users want to express in their posted images. It is extremely important for many real-world applications, such as public opinion monitoring, user abnormal behavior detection, etc. However, the automatic understanding of image implicit semantics is a research task just beginning to be explored in the literature. For understanding image implicit semantics in social media, topic detection and tracking, face attribute analysis, and communicative intent understanding are three key issues. So, this project will focus on the research of these issues. 1) For topic detection and tracking, to solve the problem of multi-modal data utilization, intuitive topic description, and massive data processing, we propose a multi-modal hierarchical topic representation model, and based on this model, we also propose one effective topic detection and tracking method. 2) For face attribute analysis, considering the properties of data in social media, we propose to do facial expression recognition and age estimation by utilizing contexts in images, which can help improve the accuracy of face attribute analysis. 3) For communicative intent understanding, since social media contains multiple types of events and persons, we propose an automatic feature extraction method instead of hand-crafted features, which can contribute to the improvement of the method's performance and universality. In summary, the methods proposed in our project can help solve basic theoretical problems in the image implicit semantics understanding, greatly improve the analyzing performance, and lay the foundation for practical applications.

社会媒体中，图像隐含语义理解的目标是理解用户发布的图像所要表达的观点、意图等，其对于舆情监控、用户异常行为检测等实际应用有重要意义。然而在相关领域，图像隐含语义自动理解的研究仍处于起步阶段。对于社会媒体中图像隐含语义理解，话题检测与跟踪、人脸属性分析、交际意图理解是其中的关键问题，因此本项目将围绕这三个问题展开研究。1）在话题检测与跟踪中，提出多模态层次话题表示模型，以及基于此模型的话题检测、跟踪算法，解决多模态数据利用，直观易懂的话题描述，海量数据处理等难题。2）在人脸属性分析中，基于社会媒体数据特点，提出上下文相关的表情识别、年龄估计算法，提高人脸属性分析的精度。3）在交际意图理解中，针对社会媒体中多事件及多人物类型的特点，提出自动的特征提取算法，改善交际意图理解方法的准确性及普适性。本项目有望突破图像隐含语义理解中关键问题的基础性理论难点，大幅度改善分析性能，为工程实践奠定基础。

社会媒体中图像隐含语义理解是对于用户要表达的想法、观点、态度和意图等的一种深度理解，对舆情分析、用户异常行为检测、信息安全监控等实际应用具有重要价值。本项目针对社会媒体中图像隐含语义理解中的关键问题开展了相关研究。1）在话题检测与跟踪方面，提出了基于互补内容感知的多模态话题检测方法，构建多模态话题图结构，在图神经网络中引入同模态的上下文和对图像和文本多对多的关联建模，有效挖掘多模态的互补信息进行话题检测。2）在人脸属性分析方面，提出了基于局部区域关系分析的人物情感识别、基于时空全卷积的连续情感识别、基于生成对抗网络的人脸表情表征及生成、基于纹理迁移的人脸图像老化表征及模拟等方法，实现了对人脸的表情、年龄等属性的有效表征及识别。3）在图像隐含的交际意图理解方面，提出了基于区域情感贡献建模及人脸分析进行图像的情感语义理解，并提出利用多模态数据进行联合推理，对图像和文字的不同层次进行表示及综合分析，实现了精确的图像交际意图理解。在本项目期间共发表学术论文17篇，其中在IEEE Transactions on Affective Computing和IEEE Transactions on Image Processing等国际权威学术期刊上发表论文9篇，在AAAI Conference on Artificial Intelligence等国际权威学术会议上发表论文8篇，申请专利4项，期间共培养/协助培养博士生5名，硕士生10名。本项目按计划完成了相关研究内容，所提出的方法有效提升了图像隐含语义理解中关键问题的识别理解性能，为工程实践奠定了基础。