基于深度多任务学习的人体行为识别研究

With the rapid development of imaging devices and internet technologies, multi-source and heterogeneous human activity data are emerged abundantly, which makes the recognition and understanding of the happened actions or events in these data more challenging. Moreover, with the scope of human activities expanded, the level of recognizing the activity is changing from the simple body part recognition to the complex social action recognition, which puts forward higher requirements for the recognition performance. This project will do the research from three aspects, including activity feature representation, multi-source data integration, and high-level semantic mining. We hope our research can break through the current bottlenecks in this field and promote the theoretical innovations and practical applications. For complicated activities, we construct a hierarchical deep model for the spatio-temporal consistence property to discovering their relation between spatial feature and temporal feature. On this basis, we study the deep integration mechanism for multi-source features to improve the robustness and discriminative ability of the activity representation. For the multi-source data, we study the deep transfer learning to deal with the problem of modal data missing, and then we build the framework of the multi-source collaborative learning to capture the intrinsic relationships among multi-modal data and to integrate the complementary information from these data. For high-level semantics, we establish multi-granularity semantic relation model and study semantic deep metric learning to help enhance the feasibility of the proposed activity recognition approaches.

随着采集设备和互联网技术的快速发展，多源异构的人体行为数据大量涌现，使得识别与理解其中发生的行为或是事件变得异常困难。与此同时，人在物理空间活动范围日益扩大，使得行为识别的目标层次从过去的简单身体局部范围识别上升到当前的复杂社交行为识别，这对行为的准确识别提出了更高要求。本项目从行为特征描述、多源数据融合、高层语义挖掘等三个方面展开研究工作，力图突破现有行为识别方法的技术瓶颈，推动行为识别的理论创新和实际应用。面向复杂行为，构建分层空时一致性关系深度模型，揭示空时特征的内在联系；在此基础上研究多源特征深度融合机制，提高行为特征描述的鲁棒性和表征能力。面向多源数据，研究深度迁移学习方法，解决模态数据缺失问题；继而建立多源数据协同学习框架，实现多源信息潜在关系抽取和互补信息融合。面向高层语义，构建多粒度语义关系表达模型，研究语义相似性深度测度方法，提升行为识别方法的适用性。

随着采集设备和互联网技术的快速发展，多源异构的人体行为数据大量涌现，使得识别与理解其中发生的行为或是事件变得异常困难。与此同时，人在物理空间活动范围日益扩大，使得行为识别的目标层次从过去的简单身体局部范围识别上升到当前的复杂社交行为识别，这对行为的准确识别提出了更高要求。本项目从空时特征融合、多源数据协同学习和语义关系挖掘三个方面开展研究。研究依赖于深度学习的视频空时信息特征融合网络，设计基于上下文调制和注意力机制的视频内容处理模型；建立多源数据深度迁移学习模型，设计多模态数据特征挖掘方法，提出多源数据深度协同学习框架；研究语义关系深度网络，设计深度联合语义一致性方法，提出底层视觉特征到高级语义的多粒度语义关系表达模型。项目在执行期间就基于深度多任务学习的人体行为识别研究在国际上发表了一系列原创成果。