面向交互式情感计算的多模态信息融合建模研究

Affective computing is an important step in human-computer interaction and intelligent monitoring that can improve the rationality for the seamless interaction. In this proposal, we study the inner mechanism between facial expression, voice tone, physiological signals and affective experience based on interactive context and multi-modal information fusion. Due to the complexity and uncertainty of affective computing, it decreases the analysis accuracy. A novel approach for affective experience evaluation in interactive environment is presented to solve the significance of those findings. From the view of cognitive intelligence, interactive environment, multi-modal information fusion, affective-factor net and affective experience distribution modal are proposed. The models can improve the analytical skills through the models' synergetic interactions. Based on affective-factor net and their cooperative interaction, multi-modal information such as facial expression, voice tone and physiological signals are applied to do synergetic dependence evaluation and to construct affective experience distribution. As well as the algorithms of multi-modal information fusion, it can improve the objectivity quality of the affective experience distribution topology including knowledge representation, causal analysis and prediction reasoning. This proposal can support for the intelligent human-computer-context interaction, and application scenarios are conducted to evaluate the rationality of the affective experience analysis approach. The affective experience with multi-modal information fusion is considered as the most individual intelligent ability for comprehending affective recognition and synthesis.

情感计算是人机交互等智能监护的难点，课题拟在交互情境中通过表情、语音语调和生理信号等多模态信息融合建模研究情感体验分布。课题采用模型化设计和形式化分析结合的解决方案，针对情感计算的不确定性和多模态信息依赖特性，提出协同依赖的情感体验与信息融合模型，具体对情境多模态信息处理、情感-动因网和情感分布的信息融和展开研究，包括：设计交互情境中动因感知和情感状态分析的形式化方法；研究情境、多模态信息和情感的内在联系，抽离表情、语音语调和生理信号等，提出多模态融合的情感-动因网模型，实现情感知识表示、因果分析和预测推理；基于情感-动因网，提出情感信息的协同依赖模型，引入多模型协同的信息融合框架，提高情感分析客观性。研究意义：提出交互情境中情感体验及其信息融合模型，为智能"人-机-情境"交互和行为决策提供理论和技术支撑；构建应用场景和系统原型，对理论模型和技术方案进行有效性和可行性验证。

课题根据研究计划进展顺利，按照基金任务书的研究目标和研究内容认真执行。课题研究交互情境中表情、语音、生理信号状态等交互情境因素和情感体验的内在联系，通过设计协同交互的信息融合框架和算法构建情感计算的体验分布模型。针对情感分析的不确定性和交互情境依赖特性，提出协同依赖的信息融合和情感分布算法，具体对情境动因、情感-动因网和情感分布的融合模型与推理算法展开研究。. 完成研究目标包括：交互情境中，表情、语音、生理信号和情境动因条件与情感状态表达的联系机制，设计情感-动因网模型和交互情境中情感状态分析和预测推理；给出情感预测结果和生理信号特征作为主、客观分析维度的广义三维情感体验分布模型和推理融合算法；基于情感体验模型及其协同依赖方法，优化情感计算的信息融合，客观、有效的计算情感体验与情境动因等表情、语音、生理信号状态的一致性分析。. 研究内容包括：交互情境感知和情感-动因网建模；基于情感-动因网的情感体验分布和多模态信息融合建模；多模型协同依赖对情感体验分布和信息融合模型的优化设计。课题研究表情、语音和生理信号的多数据信息融合对情感计算的动态性和特殊性，分析情境动因条件、交互学习和情感体验的内在联系和协同依赖关系，构建主、客观信息融合的情感体验模型和应用。. 研究意义如下：针对情感计算中“人-机-情境”的信息交互与融合和情感体验分布与协同依赖的信息融合的重要科学问题，课题提出交互情境中情感分布模型，为人机交互和行为决策提供理论支撑。分析情境信息、表情表达、语音语调和生理信号等多源信息，研究多模态信息和情感体验的因果联系，提出情感-动因网模型；设计多模型协同的融合算法，构建并优化协同依赖的情感体验分布模型，实现“人-机-情境”的智能交互。构建交互情境中的情感分析平台和应用场景，用于无人值守监控、空巢老人等的日常智能监护，为移动医疗提供辅助决策，拟最终将课题方案推广至智能家居和医疗社区，发挥重要的社会意义和工程价值。. 研究成果如下：基金第一标注，第一作者或通讯作者发表SCI检索10篇，SSCI检索1篇，单篇最高IF：3.399，获得引用11。出版“十二五”普通高等教育本科国家级规划教材1部。申请并获批2项国内发明专利，2项软件著作权。人才培养方面博士毕业2人，硕士毕业4人，博士在读3人，硕士在读7人。