基于知识与数据混合驱动概率图模型的多模态情感识别

The core issue for Human Computer Interaction (HCI) is to enable the computer to recognize, understand and express emotions, the basis of which is emotion recognition. Emotion is a kind of subjective experience, which always happens with some external changes, such as changes of facial expressions, changes of tone, rhythm, speech etc. This research will study the Probabilistic Graphical Model (PGM), based on which to fuse the information from visual and audio channels under the multilevel message passing mechanisms of human brain. The proposed method will exploit and model the prior distributions of facial activities in order to recognize facial action units (AUs), and afterwards human emotions. Domain prior knowledge is incorporated into the inference process to help reason and make decision. We will develop effective multimodal information fusion mechanism, which accounts the interactions between visual an audio channels. To handle the observation uncertainty problem, we will build a prior model to model the prior distribution and dynamics of facial actions, in which the information flow is two-way, not only bottom-up, but also top-down. Generalized prior knowledge extraction method will be investigated, which can handle various forms of prior knowledge. We will develop hybrid knowledge and data driven learning method to improve the robustness and generalization ability of the model. The successful implementation of this research will enriches the theory of pattern recognition and machine learning, and provides a new solution for multimodal emotion recognition.

人机交互核心问题是赋予计算机感知、理解以及表达情感的能力，其基础是情感识别。情感是一种内在的主观体验，但总是伴随着某些外部表现，如面部表情的变化，言语的声调、节奏和速度的变化等。本课题以概率图模型（PGM）为研究对象，模拟人脑神经元多层深度传递机理，有效地融合多模态信息，并挖掘、表示人面部动作先验分布，运用已有的先验知识来加强推理和判断，实现鲁棒地识别人面部肌肉动作单元（AU），继而识别、理解人情感状态。其中拟解决的关键问题包括：建立有效的多模态信息融合机制，解释音频信息与视频信息本质相关性；建模人面部动作先验分布及动态特性，使信息流双向流动，处理观测值不确定性问题；探求推广性强的知识抽象策略和表示方法，使其可以适用于不同类型不同形式的先验知识；发展知识与数据共同驱动模型训练算法，提高模型的鲁棒性和泛化性能。本研究的成功实施将丰富模式识别与机器学习理论，并为多模态情感识别提供新的解决途径。

本项目按照研究计划研究了基于多模态信息融合的情感识别方法、人面部动作先验分布及动态特性建模方法、以及先验知识嵌入模型训练过程方法。针对基于多模态信息融合的情感识别，本课题研究了基于语音和图像的情感识别，针对语音信号，提取了包含短时能量、基音频率、语音持续时间、前三共振峰、MFCC系数等相关统计学参数，共92维语音特征，对于图像信息，本课题分别提取语音段峰值图像的 LBP 特征以及序列图像脸部特征点的均值和标准差，在此基础上，考察了多种特征融合方法并在SAVEE数据库上测试，结果表明融合语音和图像的情感识别方法优于单一模态情感识别方法，相关成果发表PCM会议论文1篇。针对人面部动作先验分布及动态特性建模方法，本课题研究了基于动态贝叶斯网络（DBN）的AU强度识别方法，通过DBN建模AU之间的共生关系及互斥关系，以及AU强度单调变化等动态关系，显著提高了AU强度观测值准确率，相关成果发表Pattern Recognition期刊论文1篇。针对嵌入先验知识的模型训练方法，本课题研究了结构化输出标签中标签相关性嵌入模型训练过程的方法，提出了基于多标签学习的AU识别模型，将AU相关性嵌入模型训练过程，进一步提高了系统性能，相关成果发表Pattern Recognition期刊论文1篇。项目执行过程中，培养硕士研究生2名，均已毕业，博士研究生3名，目前在读。项目投入经费25万元，支出20.063万元，各项支出基本与调整后预算相符。剩余经费4.937万元，剩余经费计划用于本项目研究后续支出。