基于端到端的多语言语音内容与语言种类联合识别技术的研究

In the framework of traditional speech recognition, language recognition and speech recognition are usually regarded as two independent modules, which are cascaded to accomplish the speech recognition task of unknown languages. There are two main problems in cascade approach. One is that each module is optimized separately without comprehensive consideration, which can easily lead to error accumulation. The other is that two modules process speech sequentially, which will result in higher system delay. This project aims at design an end-to-end based joint analysis system of multi-lingual speech recognition and language identification. The joint analysis system can make use of the internal correlation between multi-lingual speech content and language category, so as to achieve synchronous recognition of language category and speech content for speech of unknown language. The joint analysis system optimizes the two tasks synthetically and integrates information at multiple levels to provide better recognition effect and reduce system delay. The main research of this project is focused on the following respects: (1) balanced fusion strategy of multi-lingual acoustic modeling units; (2) joint modeling of multi-lingual speech recognition and language identification; (3) error correction of modeling unit sequence.

传统语音识别框架下，语言种类识别和语音内容识别通常被当作两个独立的模块，二者以级联的形式共同完成语言种类未知的语音识别任务。级联方法存在两方面问题，一是各模块单独优化缺乏综合考虑，容易形成最终结果的错误累积；二是两个模块依次对语音进行处理，会造成较高的系统延迟。本项目将尝试利用多语言语音内容与语言种类信息的内部关联性，设计基于端到端方法的多语言语音内容与语言种类联合识别框架，实现针对未知语言音频的语言种类和语音内容同步识别。联合识别框架对两个任务进行综合优化，多个层面进行信息的交叉融合，从而提供更好的识别效果并降低系统延迟。本项目的具体研究内容包括：（1）多语言建模单元的平衡融合策略；（2）多语言语音内容信息与语言种类信息交叉建模；（3）建模单元生成文本序列的错误修正。

传统语音识别框架下，语言种类识别和语音内容识别通常被当作两个独立的模块，二者以级联的形式共同完成未知种类语言的语音识别任务。本项目利用多语言语音内容与语言种类信息的内部关联性，成功构建了一套基于端到端的多语言语音内容与语言种类联合识别框架，能够在完成多种语言语音内容识别的同时实现相应语音的语言种类识别。在多语言语音识别公开数据集IARPA BABEL上的实验结果表明，所构建联合识别系统语言种类识别正确率达到 99.3%，多语言语音内容识别错误率相对级联式系统下降 11.2%。本项目在执行期内按照预定计划完成所有研究内容，包括多语言建模单元融合方法、基于语言种类信息与时序特征融合的注意力机制建模方法和语言种类信息辅助的文本序列有监督生成方法。项目成果包括性能指标、论文发表和研究生培养几个方面，已全部达到预期要求。