基于多通道深度卷积神经网络的人体行为分析研究

Video-based human action analysis has wide range of applications, such as smart video surveillance, content-based video retrieval, human-computer interaction, etc. Due to the acting velocity, viewpoints and complicated backgrouds, human action analysis has been a challenging research topic. Currently, most approaches of human action analysis are mainly besed on hand-craft features, e.g., space-time interesting points and dense trajectory features. Considering the progresses of action analysis, we believe that the performance of hand-craft features is obviously limited and these features are not able to make break-through progress for video-based human action analysis. However, deep learning based methods, using original signals directly by multi-layer convolution, locally pooling and supervised feedback learning, have been largely successful on most related research fields. Focusing on the deep learning approaches of human action analysis, we mainly explore deep convolutional neural networks (DCNN) in videos. Inspiring by the theory of human visual perception that the percetions of appearance, motion and depth are relatively independent, we first propose multi-stream DCNN (MS-DCNN) based human action analysis. Considering the variation of human action would impact the training of multi-frame DCNN model, we present a coarse alignment stretage for human action frames based on multiple CNN features and dynamic time wrapping (DTW). Moreover, we propose to apply our model to action similarity labeling task with our previous work in this field. Finally, we also explore a unified framework for human pose estimation and human action recognition.

视频中的人体行为分析具有广泛的应用前景，比如智能视频监控、视频检索和人机交互等。由于行为速度、拍摄视角、复杂背景等问题，这项技术一直是研究难点。目前，人体行为分析主要是基于人工设计的特征进行展开，比如时空兴趣点和密集轨迹特征等。本项目认为，行为分析发展至今，人工设计的特征前景有限，已经很难为行为分析带来突破性进展。深度学习方法直接使用原始信号，通过多层次卷积、局部归并以及有监督的反馈学习，已在相关领域取得了巨大的成功。针对人体行为分析中的深度学习问题，本项目主要研究深度卷积神经网络模型：借鉴视觉感知中表观、运动和深度信息相对独立的理论，提出多通道深度卷积神经网络模型；考虑到人体行为的多样性对该模型多帧训练时的负面影响，提出基于多种CNN特征和动态时间规整(DTW)方法将人体行为数据粗略对齐策略；提出基于该模型的人体行为相似性验证方法；提出基于该模型的人体姿态估计与行为识别统一框架。

行为分析，包括人体行为分析和人脸行为分析，具有广泛的应用前景，比如智能视频监控、人机交互等 。项目主要研究了深度学习在视频的人体行为分析、人脸表情分析、小数据的人脸识别问题的应用。具体内容和结论如下：（1）对非深度学习的所有视频行为识别方法进行了全面总结，为深度学习方法在行为识别应用打下基础；（2）借鉴图片中物体检测的深度学习框架Faster R-CNN，提出多区域双通道的视频行为检测方法，该方法在当时所有人体行为检测数据库上达到世界先进水平；（3）为了减少深度学习对数据量的依赖，研究了一种人脸数据指数级扩充方法，使得一个小数据库能够指数级扩充进而能够正常训练深度学习网络；（4）鉴于情感分析重要性，项目研究了基于深度学习的群体情感识别和视频学生注意力识别，并在ACM ICMI国际会议上EmotiW比赛获得2个冠军1个亚军。