基于多尺度卷积神经网络模型的自发性人脸表情识别对跳转问题的研究

Zapping is defined as the action when a viewer stops watching a video by either switching channels or turning off the source. Understanding zapping behavior is a leading edge topic in human computer interaction (HCI). The analysis of zapping is limited due to the bias of traditional self-reporting questionnaire method. This project uses automatic facial expression recognition (FER) to tackle this problem. The existing FER techniques deal with posed data and focus on facial muscle motion. However, seldom of work focus on unconstrained spontaneous facial expression recognition in a more realistic scenario. In this project, Directed Acyclic Graph based Convolutional Neural Networks (DAG-CNN) is proposed to solve spontaneous FER problem. We use advertisement (ad) videos and collect users’ corresponding ad watching facial expressions, answers to questionnaire, and zapping behaviors. We then mine the relationship between the aforementioned three modalities to explore the following key challenges: (1) multi-scale feature extraction using DAG-CNN; (2) spontaneous smile intensity prediction based on DAG-CNN; (3) moment-to-moment zapping probability prediction for new users. The dataset and the preliminary results of this project have published in the IEEE Transaction, the reviewers of which agree that it is an innovative and important research.

跳转被定义为用户停止观看视频的行为。分析理解跳转行为是人机交互领域长期的热点及前沿问题。对跳转行为的研究长期受制于传统问卷反馈方法的局限性，即用户反馈时的偏见。本课题另辟蹊径，提出了以情感计算为理念、基于表情自动识别的方案解决跳转预测问题。现有的表情识别方法约束用户头部姿态固定，仅关注面部表情引起的肌肉运动形态。在面向真实的环境下，非约束的自发性人脸表情识别的研究还处于起步阶段。本课题提出无回路有向图框架下的多尺度卷积神经网络（DAG-CNN）深度学习模型针对非约束的自发性表情识别，结合用户观看时的表情视频、偏好反馈问卷、以及跳转行为三类数据，建立跳转模型，拟解决如下关键问题：（1）基于DAG-CNN的多尺度特征提取；（2）基于DAG-CNN的自发性笑容强度动态预测；（3）基于新用户表情的跳转概率动态预测。本课题预研结果已被IEEE Trans收录，评审一致认为跳转预测问题新颖且重要。

本结题报告深入探讨了在研究“跳转问题”中的两个问题：1. 实时基于流的混合运动对齐问题。2. “跳转问题”的数据库公开发表与基线结果。运动物体通常包含刚体和形变的表面（比如人脸）。于是，在视频分析任务中，在物体本身运动的时候，跟踪和测量物体表面形变，是一个重要且挑战的问题。比如，基于视频的表情分析需要在头部经历刚体运动的同时，跟踪非刚体的肌肉运动。在这篇报告中，我们提出了一个视频按帧对齐的通用模型，该模型的推到利用了贝叶斯框架，我们从理论上证明了这个模型在一些假设下，可以转化成基于SIFT流和光流的最小二乘法问题。我们也提出了动态规划可以用来加速我们的算法以达到实时处理。