基于视听信息融合的情感机器人情感识别与情感建模研究

With the rapid development of the information age, the Intelligent machine – human interaction has become the main trend and the urgent need for Human-computer interaction, emotion recognition and expression is the key technology. However, most of the existing approaches focus on a deliberately displayed series of exaggerated affective expressions, these approaches that are aimed at recognition of a small number of prototypical (basic) expressions of emotion , and single-modal approaches, where information processed by the computer system is limited to either face images or the speech signals. Consequently, they are unable to catch frustrated or depressed feelings of users, and cannot response properly according to users' emotional states. In this proposal, we focus on the recognition of user facial emotional and speech emotional states in emotion robot system, which is the funder mental step for intelligent emotion Human-computer Interaction. This proposal specifically includes: User emotion analysis and representation in Human-computer interaction. By coding of multi-dimensional emotions, the links between the basic emotions are established; Human-computer interaction emotion feature analysis, building a visual and/or audio spontaneous emotion databases which contains multi-source emotion features and users' real feelings; For facial and speech expression, research on the algorithm of intra-class variation sparse representation and public sparse representation of the visual-audio spatial and temporal emotion features. Research on the fusion theory and methods of visual and auditory emotional features based on sparse representation and multilayer structure theory for visual-audio emotion classification, proposing modeling the context of time series. Research on the affective computing model for emotional robot, proposing modeling user emotion sequence Based on the mental state transition network by introducing emotion driving strength ,personality characteristics and different implicational fuzzy controller , an emotional state transition model will be constructed which fuse the facial expression emotional energy, speech emotion emotional energy and the state of emotional state transition network, and to reflect robot'emotional overlay and decay. Research and realize a human-computer interaction system based on an emotional robot. Research on these issues will provide new ideas and solutions for user emotion recognition and robot emotion modeling, and lay the theoretical and technical basis to promote Human-computer Interaction.

智能情感人机交互技术是高速信息时代人机交互发展的必然趋势和迫切需求，情感机器人情感识别和建模是其战略性关键技术。而目前研究主要是针对人为的离散情感进行识别和建模，很少涉及对人类自发表情和语音情感序列的分析以及机器人个性化的情感应对。本课题致力于研究融合表情和语音的用户情感序列识别和机器人情感建模问题。研究内容包括：用户自发表情和语音等情感信息的获取及情感数据库的构建，提出基于半监督学习的情感库标注方法；用户多维情感的形式化表示方式，建立基本情感之间的联系；基于感知机理的用户表情和语音情感时空特征提取及稀疏表达方法，构建用户情感序列识别模型，提出基于表情和语音情感的协同识别模型；构建情感机器人的个性化情感状态转移模型，反映人机交互过程中机器人的情感叠加与衰减。该项目的研究不仅能满足智能人机交互中的人机情感识别和建模技术的迫切需求，而且能为人机情感交互发展打下一定的理论和技术基础。

本项目围绕面向人机情感交互的机器人情感识别和建模关键技术开展深入研究。通过理论分析、计算模型构建、搜集社会媒体样本、自建样本、试验场景设计与实验验证等手段，致力于解决融合表情和语音的双模态情感识别和机器人情感建模中的关键技术和科学问题。主要研究内容包括：（1）基于项目组已有的会话情感语料库、语音情感库和人脸表情库，构建了一个包括不同种族、不同性别、不同年龄阶段的自然场景下的人脸表情数据库，提出了一种基于卷积神经网络的多任务投票机制的图像自动标注（MVAIACNN）方法；（2）针对表情和语音的时空特征提取及表达，构建了中心对称局部平滑二值模式CS-LSBP、局部纹理描述子CS-LOP、中心对称局部符号幅值模式CS-LSMP、方向幅度特征图OMFMs、正负幅度特征图PNMFMs和绝对梯度方向直方图HOAG。针对表情图像和语音数据的时空对齐问题，提出了基于深度学习的表情图像序列和语音情感时空特征及深层语义特征的提取方法；（3）针对基于表情和语音的多层次用户情感协同识别模型，提出一种滑动窗口动态时间规整算法以自动选取视频中表情表现明显的图片序列；融合局部强化运动历史图，借鉴LBP-TOP原理，提出具有时空域描述能力的时空韦伯局部描述子STWLD来提取动态纹理信息；构建了多级注意力多任务学习模型进行连续模型下的人脸表情识别；提出了基于知识蒸馏的视频-音频双模态情感识别方法；（4）针对机器人的情感状态转移模型及情感序列建模，提出一种基于能量守恒原理的机器人前向机械模型，构建了反映电机连续运动的时序预测模型。为提高机器人表情生成的时空相似度和运动平滑度,将底层特征语义与高层情感语义融合作为动态表情的时空特征，提出一种基于双LSTM（长短期记忆）融合的类人机器人实时表情生成方法。在本项目的资助下,共发表37篇学术论文，其中SCI论文16篇。获得授权发明专利4项。2018年获中国发明协会发明创业成果奖一等奖（国科奖社证第0123 号，证书号：2018-CAICG-1-B05-2）。. 本项目培养硕士25名、博士2名，目前19名硕士生取得硕士学位和1名博士生取得博士学位。培养青年教师1名。. 本项目的研究成果为智能人机情感交互系统中的人类情感识别的研究提供了新的思路和解决方法，为人机情感交互发展打下了一定的理论和技术基础。