基于超声图像的静音语音识别关键技术研究

Speech recognition based on articulator movement (silent speech recognition)can be applied widely in many areas, such as speech recognition for patients with laryngectomy;making calls but speaking without sound in the situations like venue and cinemas where we need to keep silent or privacy,speech recognition in high-noise environments and so on. This project will use ultrasound machine and cameras to acquire images about speaker's articulator and recognize articulator's movement to get corresponding speech by the images. The research will focus on several key technical points about silent speech recognition. First, we will use constrained Boltzmann machine and unsupervised learning to extract features and decrease dimension nonlinearly for images about lips and tongues movements recorded by camera and ultrasonund machine,respectively. The research uses multiple constrained Boltzmann machines to build a deep neural network to fuse multi-channel and multi-modal data.We will use hidden Markov model in silent speech recognition by the characteristics derived from the fusion of multi-modal data.In the research, there will be innovative achievements in the feature extraction, multi-channel data fusion, articulator recognition and so on.

基于发音器官的运动来识别语音（静音语音识别）可应用到广泛的领域，如识别喉切除的病人说话;在会场及电影院等需要保持静音或者隐私的场所不出声音只运动发音器官来打电话;再比如在高噪声环境中进行语音识别等。本课题将用超声仪结合摄像头来采集说话人的发音器官运动，基于唇部运动图像及舌头运动图像来获取发音器官运动,从而识别相应的语音。本课题将基于实现静音语音识别中的几个关键技术点来展开研究。首先，将利用有限波尔兹曼机通过非监督学习对摄像机记录的唇部运动图像及用超声仪记录的舌头运动图像分别进行特征提取及非线性的降维。为了实现多模态发音运动数据的融合，本课题利用多个有限波尔兹曼机搭建一个深度神经网络来对多通道、多模态的发音运动数据进行数据融合研究。基于多模态融合获得的发音运动数据特征，利用隐马尔科夫模型进行静音语音识别研究。本课题将在发音运动特征提取、多通道数据融合、发音运动识别等方面有创新性成果。

本项目通过三年的研究工作，已完成项目计划基本内容，并基于最初计划进行拓展研究，结合目前热门算法及研究方向，对研究内容进行改进。本项目不仅采集了计划内汉语数据库，并对藏汉双语数据库进行设计及采集工作。基于采集数据，本课题完成了数据分析、标记、处理，静音语音识别研究以及测试、评价、优化等工作。基于深度神经网络实现了对超声图像的发音运动识别，从而实现了静音语音识别。再此基础上基于DNN实现了发音运动与语音信号间的映射关系，实现了初步的发音运动可视化。发表CCF A类会议IEEE VR文章1篇，SCI期刊论文6篇，CCF B类会议ICASSP1篇，CCF C类会议Interspeech1篇。