面向大规模视觉语音信号的复杂情感发现与分析方法研究

While most research on automatic affect sensing and recognition has mainly focused on analyzing facial and vocal affect in terms of a small number of basic emotions, in this project, we aim to detect complex emotion and explain its intimate relationship with cognition in the big data context. Complex affective states are not independent from one another; rather, they are related to one another in a systematic manner. Thus, it is essential for us to develop novel statistical learning and data mining models by leveraging the interdependencies among all the emotional states. .The overall objective of the proposed research is to develop innovative algorithms that can detect emotion expressions by analyzing facial expressions and speech. This will improve the quality and fidelity of human computer interaction. The research seamlessly integrates deep learning, ranking algorithm, graphical models and transfer learning, and will be conducted as a coherent sequence of four tightly coupled phases: (1) To exploit weakly-labeled relative-order information among the emotion labels in the continuous dimensional space. Inspired by the recent advances of ranking feature learning, we propose ranking-Convolutional Neural Network (ranking-CNN), a ranking-based novel feature learning algorithm to learn robust, discriminative ranking features for emotion recognition and discovery. We also propose a novel algorithm to make ordinal decisions and then aggregate the results of ranking-CNN to make the final ranking prediction in the VA space. (2) To exploit intuitive emotion topic feature and discover new emotion. We introduce the hierarchical Bayesian model to learn emotion topic, and then project these emotion topic to the VA space by integrating the output of ranking-CNN. We also propose a novel method to classify the emotion topic into four categories (weakly-labeled topics, matched topics, new emotion topics and emotion-irrelevant topics), and then with hierarchical ranking, new emotions are discovered. (3) To develop a unified, self-adjusting framework to detect and recognize complex emotional expressions. Capitalizing on the recent advances on transfer learning, we propose a weakly labeled Gibbs sampling algorithm in Latent Dirichlet Process (LDA) that is driven by general feature regularities while sharing the semantic regularities of interest among the emotion classes. More specifically, an updater is built for each emotion class, and the semantic correlations between the models are leveraged from one class to another, from labeled basic emotions to unlabelled complex emotions through feature-level transfer learning and parameter sharing. Our model can account for both shared emotion themes and unanticipated emotion. (4) To develop a software system for emotion detection, discovery, and analysis in psychological consulting and therapy. We have collected a huge amount of audio and video recordings in therapy sessions, based on which, we will evaluate our algorithms and develop a software system for emotion analysis. Our aim is to identify predictive emotions associated with behavior changes, counter-changes and ambivalence.

针对大数据背景下复杂情感类别无法预先确定，大部分样本缺乏标记，情感间关系未充分利用的问题，引入概率图模型、排序比对、深度学习和迁移学习理论，研究实际场景下，视觉或者语音信号中复杂情感发现、特征学习和迁移以及准确识别的问题，并将成果应用于心理咨询中。具体研究内容包括：研究新的排序深度卷积神经网络用于学习体现情感间关联且具有可鉴别性的排序比对情感特征，并研究排序比对结果的聚合方法准确识别情感；研究概率图模型用于学习直观的情感主题，通过分析情感主题在情感空间的分布准确发现新的情感；研究在大数据基础上，借助迁移学习，如何将情感特征迁移到新的情感类别和无标注的数据；最后，针对心理咨询的实际场景，分析咨询者和心理医生的情感状态，为心理辅导提供有力支持。该研究为基于连续维度的复杂情感计算提供了很好的思路，可推动真实环境中连续情感计算和新型人机交互技术的发展，具有很好的理论意义、实际应用价值和社会效益。

本项目针对自然环境下带标记情感样本不足，情感间关系未充分利用的问题，引入排序比对、深度学习和迁移学习理论，研究自然场景下，视觉或者语音信号中细微情感识别，情感特征学习和迁移以及准确识别的问题。主要研究内容包括：研究自然环境下人脸表情图片生成、特征学习与识别方法，借助生成对抗网络，提出基于多任务协同的鲁棒人脸表情识别方法；研究弱监督web图片情感识别方法，利用注意力机制以及弱监督方法进行情感区域定位和嘈杂的web图片情感识别；研究传统光流与深度学习结合的微表情自动识别方法，解决微表情中数据量少而带来的过拟合问题，提高微表情识别率。研究无监督域适应语音情感识别方法，提出了基于三重注意力的非对称卷积神经网络语音情感识别的方法，解决语音识别跨域泛化性能差的问题；研究并提出基于监督变分编码器的混合语音事件可变因子学习方法，提升混合语音事件识别率；研究基于排序的多模态连续情感识别问题，提出了基于单样本有序回归的维度情感识别方法，以及基于层次注意力机制的多模态维度情感识别方法，挖掘视觉语音信息之间的关联性，提升多模态识别效果；研究情感感知的图像问答，提出了多情感感知的图像和视频问答生成方法。在本项目支持下，发表论文25篇，授权发明专利4项，培养博士3名，硕士12名，其中1名博士生获得江苏省优秀博士论文。成果转化后应用于精神病人辅助诊疗、驾驶员疲劳监测、在线学习者学习状态监控等领域，获得了很好的经济效益，获得教育部高校科学研究优秀成果二等奖、江苏省科技进步三等奖、中国公路学会科技进步一等奖各1项。