基于深度时空关系的视频关键点跟踪

Facial landmark tracking is a fundamental problem in face analysis. It has been widely applied in face recognition, augmented reality and human-computer interaction. Although image based facial landmark localization has achieved great success, it is still a challenging problem when the landmarks should be tracked in videos. The efficiency on handheld devices, the accuracy in unconstrained environment and the stability in long-term tracking are basic problems for landmark tracking. The lack of data makes the task more difficult. Based on deep learning, this project aims to improve the landmark tracking performance in unconstrained videos. Specifically, we design a low-cost network architecture to improve the efficiency and exploit the spatial-temporal contextual information to improve the accuracy. A new loss function which optimizes strong and weak semantic landmarks respectively is proposed to further mine the landmark semantic positions from labelled data. To make the image based training data applicable on landmark tracking, we synthesize the virtual previous frame for each image so that it is tuned into into a short video, which significantly expands the training data. Finally, we novelly investigate landmark labeling and propose a semi-automatic labeling method, which remarkably reduces the difficulty of labeling dense landmarks. The research in this project will advance both the research and the applications of facial landmark tracking.

人脸关键点跟踪是人脸分析领域的一个基础问题。目前基于单帧的关键点定位在多个数据库上已经取得了较好的性能，然而基于视频的关键点跟踪面临着一系列难点问题。低功耗设备上的高效率、复杂场景下的精确性和长时跟踪的稳定性是关键点跟踪的三大难点。数据的缺乏进一步加剧了算法设计难度。本项目在深度学习的基础上对上述难点问题展开研究，首先通过分析视频中的时空上下文关系，设计了时空特征融合、高性能低功耗网络以及并行关键点置信度三个模块，同时提高了跟踪的精度、速度和稳定性。其次，在训练策略方面，将关键点根据语义特征分为强语义和弱语义点并分别设计独立的损失函数，充分挖掘了网络的拟合性能。再次，通过研究基于三维技术的虚拟前帧生成算法，将单张图像数据虚拟为极短视频，增广了训练数据。最后，通过对稠密关键点标注方法的研究，将标注由手动变为半自动，极大减少了标注工作量，为获得大量关键点数据提供了基础。

本课题针对关键点定位中低功耗设备的高效率、复杂场景下的精确性、长时间跟踪的稳定性以及数据缺乏等难点问题展开研究，研究开发了基于深度时空关系的关键点跟踪技术，包括：1) 基于时空上下文的深度关键点跟踪框架；2) 基于非刚性曲线拟合的损失函数；3) 基于三维人脸模型的前帧图像虚拟生成和 4) 基于人脸结构的半自动稠密关键点标注。相关技术在国际公开测试集上均取得了同时期国际领先的结果，相关工作整理发表在国际顶级期刊和顶级会议上。本项目公发表论文18篇，申请发明专利2项，项目成果有效推动了人脸关键点跟踪技术的发展，具有较好的实用价值。