面向语音与面部表情信号的多模式情感识别技术研究

Emotion recognition is the key scientific issue that must be solved in human-computer interaction. This project takes speech signal and facial expression signal as the research subjects, takes the extracting of features, the fusion algorithm of features and recognition algorithm as the research contents, and finally accomplishes recognition for six kinds of human emotion (joy, anger, surprise, sadness, fear, neutral). The marked characteristics of this study are: 1) A method which extracts speech emotion dynamic feature adaptively is proposed. The method improves the robustness, because it unites the time-frequency agglomeration performance for modulation spectrum and less sensitive to noise for MUSIC spectrum estimation. 2)A method which extracts facial expression feature by uniting minus model and PCA is proposed. The method reduces computation efficiently, has the ability to extract slight change of image. The ability chimes in with the characteristic that facial expression information is mainly reflected in the local. 3) Fusion algorithm based series-parallel combination (FASPC) is proposed. The method improves the accuracy of emotion recognition by giving full play to the advantages of decision level fusion and feature level fusion, and makes the whole fusion process close to human emotion recognition more. Research results will provide new methods into multimodal emotion recognition, promote the further development of human-computer interaction system, make a significant contribution to harmonious development of our society.

情感识别是人机交互领域中必须解决的关键科学问题。本项目以语音和面部表情信号为研究对象，以情感特征参数的提取和参数融合与识别算法为研究内容，实现对高兴、愤怒、惊奇、悲伤、恐惧和中性六种人类基本情感的识别。其显著特点是：1）充分利用了调制谱的时频集聚性和MUSIC谱对噪声敏感度低的特点，提出一种自适应的语音情感动态特征提取方法，提高了特征参数的鲁棒性；2）提出采用差值模版与PCA相结合的技术提取面部表情特征，有效地降低了运算量，且具有提取图像局部细微变化的能力，与面部表情信息主要体现在局部的特点相吻合；3）提出一种串并结合的多模式信息融合方法FASPC，充分发挥了决策层融合与特征层融合的优点，使整个融合过程更加接近人类情感识别，从而提高了情感识别的准确性。研究成果将为多模式情感识别提供新的研究方法，促进人机交互系统进一步深入发展，对我国社会和谐发展具有重要意义。

在人机交互中需要解决的问题实际上与人和人交流中的重要因素是一致的，最关键的都是情感智能的能力，计算机要能够更加主动地适应操作者的需要，首先必须能够识别操作者的情感。项目设计了一套基于语音和面部表情信号的多模式情感识别方法，充分利用了调制谱的时频集聚性和MUSIC谱估计对噪声敏感度低的特点，提出了一种自适应的语音情感动态特征提取方法，提高了特征参数的鲁棒性；在Bagging算法和AdaBoost算法的启发下，提出了一种串并结合的多模式信息融合方法FASPC，充分发挥了决策层融合和特征层融合的优点，使整个融合过程更加接近人类情感识别，从而提高了情感识别的准确性；在FASPC算法中，选择了两种不同类型的分类器，使个体分类器之间差异变大，集成效果变好。并通过双误差异性选择策略挑选出大于平均差异性水平的分类器用来做最终情感识别，降低了计算复杂度，提高了精度。该方法将对情感的研究从感性认知上升为可计算模型，对探索大脑对信息的分析和处理机理、进一步加深对大脑中情感概念的解析和理解，不但在理论上是可行的，而且可以形成富有特色的一类情感识别技术。其研究成果对提高我国工业产品的科技含量，建立良好和谐的人机交互环境，形成“感性工业”产业链起到积极意义。