基于时空特征融合的连续手势识别研究

Human gestures are continuous in natural human computer interactions, and the boundaries between different gestures are unknown. Gesture recognition based on wearable devices restricts the freedom and naturalness of gestures, and is expensive. The segmentation and recognition accuracy is limited, for the vision based gesture recognition methods which segment and recognize gestures consecutively. In order to overcome the problem, this project focuses on the research on the vision based continuous gesture recognition based on the extraction and fusion of spatiotemporal features using deep neural networks: First, 3D convolutional neural networks and convolutional long-short-term-memory recurrent neural networks are utilized to extract the low-level short-term spatiotemporal features and high-level long-term spatiotemporal features respectively, for the construction of spatiotemporal feature maps of gestures. And then, the multi-scale segmentation generation and the punishment strategy of detection error are researched, for continuous gesture dynamic segmentation and recognition. Lastly, the context model of gestures is utilized for the recognition optimization. Continuous gesture recognition is researched in-depth and optimized according to the data characteristics and application requirements in this project. The research will play a positive role in promoting the gesture recognition applications, such as human-computer interaction, and sign language translation.

在自然人机交互过程中，人体手势是一个连续的行为过程，不同手势之间的边界未知。基于可穿戴设备的手势识别限制了手势的自由性和自然性，且价格昂贵。基于视觉的先分割再识别的连续手势识别算法的分割与识别精度不足。针对此问题，本项目以基于深度神经网络的手势时空特征提取与融合为基础，对基于计算机视觉的连续手势识别展开研究：首先，利用三维卷积神经网络和卷积长短时记忆循环神经网络，分别提取手势的低层次短时时空特征和高层次长时时空特征，研究手势的时空特征图的构建方法；然后，基于手势的时空特征图，探索多尺度片段生成机制及检测边界错误惩罚机制，研究连续手势动态分割与识别方法；最后，研究利用不同手势间的上下文模型，对连续手势的识别结果进行优化，实现高精度连续手势识别。本项目层层深入地对高精度连续手势识别展开研究，不断根据数据特点和应用需求进行算法优化，对高精度人机交互、人机手语翻译等手势识别应用起到积极的推动作用。

手势是自然人机交流的一个重要交互方式，本项目分别从手势时空特征图构建、连续手势动态分割与识别、连续手势识别优化等方面开展连续手势识别研究。首先，提出了基于三维卷积神经网络和卷积长短时记忆循环神经网络的手势时空特征图提取模型，并对卷积长短时记忆循环神经网络的冗余性和注意力机制进行了探索，提出了新的网络变种。然后，提出了连续手势的动态分割与识别方法，利用时域膨胀Res3D网络进行片段分割，利用三维卷积神经网络和卷积长短时记忆循环神经网络架构进行片段识别。最后，提出了连续手势的同时分割与识别方法，并利用图卷积对基于骨骼的行为序列进行识别，提出了拓扑可学习的自适应图卷积方法，提升了识别效果。本项目所提出的方法，可以直接应用到基于多模态数据（RGB、深度、骨骼）的自然人机交流当中，提升人机交互的效率。