面向智能人机交互的移动机器人双耳声源定位研究

Mobile robot auditory system is an important way for friendly interaction between a robot and external objects. For binaural sound source localization, the direction and distance of a sound source can be estimated by the perceptive differences of two ears of the robot to the sound source. However, the movement of a mobile robot will lead to continuous changing of the mounted dummy head 's position and posture, which brings about many new challenges to the existing approaches. As with a moving robot, it is difficult to extract effective binaural cues in complex environments. High computational complexity of present multiple sound sources localization algorithm can’t satisfy the real time requirement in continuous localization. Signals received by a microphone array will own inconsistent spatial constraints due to the movement and rotation of the dummy head. This project researches the dummy head topology which can be conveniently carried on various mobile robot platforms. For a real-time and robust multiple sources localization method, binaural microphones based clustering triangulation localization algorithm is used. Then, a dynamic cross-correlation warping technology is introduced, which can effectively estimate the time delay and intensity from the microphone signals with inconsistent spatial constraints. By using the spatial dependence of signal delay and joint analysis the time difference and intensity difference within narrow/broad frequency band, an information fusion technology is designed for multiple signals emitted by the identical or different sound sources. The effectiveness of the proposed sound source localization method mentioned above will be tested by many offline and online experiments in HRI environments including mobile robots and speakers.

移动机器人听觉系统是实现机器人与环境目标自然、友好交互的重要途径。双耳声源定位方法利用两只麦克风对同一声源信号的感知差异实现声源方位和距离的计算。人机交互面临复杂的噪声、混响等声学环境，机器人本体运动会造成机载双耳人工头空间位置和姿态的连续变化，对现有的定位方法和系统形成了新的挑战——复杂环境下双耳定位线索难以提取；多声源定位算法复杂度高难以满足机器人运动中连续定位的实时性要求；机器人本体运动会导致麦克风阵列接收声源信号空间约束的不一致。本项目研究具有一定抗噪、抗混响能力的双耳麦克风阵列实时鲁棒的声源定位方法；提出动态双耳线索规整方法，从不一致的信号空间约束中提取有效的时延和能量信息；基于信号时延空间和时间依赖关系，建立机器人运动过程中运动声源定位的信息融合策略。通过移动机器人对周围说话人的声源定位大量实验，验证上述方法的有效性。

移动机器人听觉系统是实现机器人与环境目标自然、友好交互的重要途径。双耳声源定位方法利用两只麦克风对同一声源信号的感知差异实现声源方位和距离的计算。人机交互面临复杂的噪声、混响等声学环境，机器人本体运动会造成机载双耳人工头空间位置和姿态的连续变化，对现有的定位方法和系统形成了新的挑战——复杂环境下条件下双耳定位线索难以提取，多声源定位算法难以满足机器人运动中对移动声源连续定位的实时性要求，交互场景下声源间歇性发声会导致活跃声源数量动态变化、声源位置跟踪失败。.本项目围绕智能人机交互场景下的移动机器人双耳声源定位展开研究。针对噪声混响条件下双耳定位线索受到严重影响问题，提出一种基于混响加权机制和广义参数匹配方法来抑制前后混响同时保留直达信号双耳线索，采用深度卷积神经网络提取噪声无关的双耳互相关线索进而用于时延估计；针对多声源定位实时性要求和声源个数未知带来的性能下降问题，提出时延补偿估计算法，打破了双耳时间差和能量差线索提取不同步的局限性，联合双耳线索聚类分析策略实时定位多声源，设计一种基于空间主特征向量的多声源个数和位置联合估计算法，实现活跃声源数量动态变化条件下的多声源定位；针对机器人运动场景下的移动声源连续定位，在粒子滤波器中加入观测值引导机制和视听多模态融合策略，利用当前帧位置信息和视觉人体身份信息来辅助完成间歇性发声声源的连续跟踪；针对实际人机交互场景的友好型需求，提出基于互注意力迭代机制的特征对齐网络多模态关键词识别算法启动人机对话，通过融合同一声源多次发声的信息完成最终的定位决策。.本项目搭建了支持双耳听觉的移动机器人实验平台，建立了机器人双耳声源定位实验仿真和真实场景数据库，通过进行多场景、复杂条件下的声源定位性能测试验证上述方法的有效性和先进性，达到了预期的技术和成果指标。