中英文混合语音识别中声学建模关键技术研究

With rapid advances of intelligence speech technologies and the growing international trend of our communication, techniques to obtain excellent Mandarin-English code-mixing speech recognition performance are highly desired. However, conventional related research usually focuses on the inter-sentence bilingual speech recognition cases, and this makes the technology difficult to meet the needs of most users today. In this proposal, we focus on the acoustic modeling technologies for Mandarin-English mix-lingual speech recognition with intra-sentence code-mixing. More specifically, the mix-lingual acoustic modeling techniques based on the deep neural networks will be investigated. Our solution involves three key techniques, including: (1) the data augmentation techniques are proposed to enhance the acoustic environments and to increase the amount of the Mandarin-English code-mixed speech data; (2) the proper mix-lingual lexicon construction and correction approaches are introduced to eliminate the negative effects from the differences between the Mandarin and English phonemes. New approaches are originated from the combination of feedback information of the speech recognition errors, the knowledge of linguistics and phonetics, and the conventional data driven algorithms for phone mapping; (3) the knowledge transfer learning methods are investigated for transferring the linguistic and non-native accent nature of the embedded English speech data. New algorithms for deep feature representation and mix-lingual acoustic model adaptation will be carefully designed. Finally, we will combine all of the key techniques to build a Mandarin-English mix-lingual speech recognition demonstration system to validate our proposed techniques. Furthermore, we plan to cooperate with some relevant enterprises to promote the application values of our Mandarin-English code-mixing speech recognition system.

随着智能语音技术的快速发展以及人们沟通交流的国际化趋势日益加剧，高性能的中英文混合语音识别技术成为迫切需求。然而，传统的中英文语音识别大多注重于研究中英文句间切换的情况，这使得其难以满足当今大多数用户的需求。本项目致力于研究中英文在句内混用的语音识别混合声学建模技术，建立统一的技术框架，针对其中的关键科学问题展开深入探讨。具体内容包括：对中英文混合语音数据量及其声学环境进行增强，提高声学模型深度学习统计建模的准确性和鲁棒性；提出结合语音识别错误中的反馈信息、语音学知识和已有的数据驱动算法，构建更合理的中英文混合发音词典，消除中英文音素差异带来的负面影响；从深度特征表达和声学模型自适应两方面实现非母语英语口音特性和英语语言属性的知识迁移学习，提高混合声学模型对英文发音变异的鲁棒性；搭建一套中英文混合语音识别示范系统，并与相关企业合作进行成果推广及应用。

中英文混合语音识别是指待识别的语音中同时包含中文普通话和英语的语音识别。随着智能语音技术的快速发展以及人们沟通交流国际化趋势日益加剧，高性能的中英文混合语音识别技术成为迫切需求。本项目着重研究了在以中文普通话为主体语中嵌入英语的自动语音识别声学建模中的关键科学问题。主要内容包括：(1)借鉴半监督和主动学习思想，提出了基于异构深度神经网络架构的自动标注算法，实现了大规模多领域的混合语音片段自动标注，对中英文混合语音数据量及其声学环境进行了增强，提高了声学模型深度学习统计建模的准确性和鲁棒性；(2)提出了一种基于声学数据驱动的混合语言发音词典修正方法，同时结合语音识别错误中的反馈信息、语音学知识，构建更合理的中英混合发音词典，在一定程度上解决了非母语英文口音发音变异问题，消除了中英文音素差异带来的负面影响；(3)提出了与语言相关的深度表征抽取方法和基于预训练的端到端混合注意力机制的声学建模方法，大幅度提升了整体语音识别系统的声学建模能力，提高了现有混合语音识别系统的性能; (4)积极与工业界进行产学研合作，阶段性研究成果及时应用到工业界相关产品中，取得了较好社会价值和经济效益，并在此基础上成立校企联合实验室，为智能语音领域协同培养高质量技术人才。综上，本项目的研究方法和技术对混合语音识别的研究起到重要的推动作用，为构建大规模中英文混合连续语音识别应用引擎提供了理论支撑和实践经验。