融合表面肌电和加速度信息的中国普乐手语识别研究

The realization of continuous sign language recognition system has important significance and broad application prospects. The Chinese Pule Sign Language, proposed by the Deaf Association, is combined with the Chinese phonetic rules and the Chinese character structures. They are easy to understand, regulate and execute and the coded gesture collection has a small and constant scale, which has potential advantages in vocabulary scalability and user-independent sign recognition tasks. This study aims to explore the feasibility of Chinese Pule Sign Language recognition, especially for user-independent recognition, based on the fusion information of surface electromyography and accelerometer signals. Firstly, the complementary superiority of using two types of sensors is taken to detect gestures, and the schemes of gesture normalization and sensor configuration will be proposed to improve the uniformity and separability of gestures. Secondly, the entropy theory, characterizing the complexity and random uncertainty of time sequences, and peak detection algorithms are employed to achieve effective segmentation of continuous gestures. Meanwhile, the bilinear models are used to extract two independent factors of feature vectors, from which the motion-dependent factor will be chosen as user-independent feature to involve in user-independent feature reconstruction. By this means, a satisfactory and robust scheme of user-independent sign language recognition will be realized. Finally, the order of identifying each element, as well as the extrapolation ability of bilinear models will be studied to verify the probability of reducing users’ training burden. The study will provide a new thought and supplementary form for the realization of large scalable and user-independent sign language recognition system. On the one hand, the research achievement will directly benefit the deaf society in promoting the standardization process of sign language. On the other hand, it will help the deaf society to communicate fluently and can better integrate into the hearing society.

连续手语识别具有重要研究意义和广阔应用前景。聋人协会提出的中国普乐手语，易于理解、规范和执行，其规模较小且数目恒定，在词汇量可扩展和非特定人手语识别方面具有潜在优势。据此，本课题重点探索融合表面肌电和加速度信息的中国普乐手语识别，特别是非特定人识别技术。首先充分发挥两类传感器检测手势动作的互补优势，提出动作规范方案和电极配置方案，提高动作的统一性和可分性；其次利用表征信号复杂度和随机不确定性的模糊熵理论，以及峰值检测等算法，实现连续手势动作的有效分割；之后采用双线性模型进行因素提取和用户无关特征重构，实现非特定人手势动作识别。最后从要素识别先后顺序和发挥双线性模型外推能力的角度，研究用户训练负担减小的可行性。本课题为实现词汇量可扩展的非特定人连续手语识别系统提供新的思路和补充形式，其研究成果将直接造福于广大听障人群，有助于推进手语标准化进程，提高听障群体受文化程度以及与健听人的顺畅交流。

本项目针对聋人协会提出的中国普乐手语，由于其易于理解、规范和执行，且其规模较小、数目恒定，在词汇量可扩展和非特定人手语识别方面具有潜在优势的特点，充分利用表面肌电电极和加速计在捕获精细手势动作和大尺度动作轨迹方面的互补优势，提出了普乐手语动作规范方案和电极配置方案，提高了动作的统一性和可分性；其次利用表征信号复杂度和随机不确定性的熵理论，实现了基于样本熵的连续手势动作的有效分割；之后采用双线性模型进行因素提取和用户无关特征重构，实现了非特定人手势动作识别。最后从要素角度出发，提出了包含223个中国汉字的连续普乐手语手势动作识别研究，为用户训练负担减小提供了可行性，为词汇量可扩展的连续手语识别系统提供了新的思路和补充形式。本项目的研究成果将直接造福于广大听障人群，有助于推进手语标准化进程，提高听障群体受文化程度以及与健听人的顺畅交流。本项目共发表论文6篇，其中SCI论文2篇，EI论文3篇，获得青年论文优秀奖1篇；申请国内发明专利2项。指导硕士研究生3名，与国内外多个知名研究机构保持学术合作与交流。