基于多模态协同学习的智能轮椅室外三维环境认知与安全预警

Based on the physiological and psychological features of the old and handicapped people, this project uses the multi-modal information of point clouds, images, and sounds to research the 3D outdoor environment cognition and safety pre-warning techniques of the intelligent wheelchair. These techniques can help the old and handicapped people to cognize 3D outdoor environments, warn dangerous conditions, and guarantee their comfortable and safe going out. In this project, a 3D laser colored scanning and range detecting system is developed for the intelligent wheelchair to obtain the 3D colored point cloud data and build the 3D colored geometric model of the outdoor scene. Then, the principal component features and color features are constructed, and a multi-level segmentation hierarchy is designed to partition the 3D colored point cloud of the outdoor scene accurately based on the multi-modal features of the point cloud and image. By using the multi-modal advantages and unlabeled information, a 3D colored point cloud classification method is proposed to construct the semantic description of the outdoor scene for the intelligent wheelchair based on the multi-modal collaborative learning of the point cloud and image. In addition, a bionic touched sound sensing system in the front of the intelligent wheelchair is developed, and a pavement material recognition method is proposed based on the multi-modal collaborative learning of the image and sound for the intelligent wheelchair. Based on the 3D outdoor environment cognition, the collision and instability pre-warning models are constructed for the intelligent wheelchair, and the safety pre-warning mechanism of the intelligent wheelchair is built to forecast the safety danger. This research not only has the important social significance and extensive market prospect, but also has the stronger theoretical significance for enriching the humanlike perception and cognition techniques and the service robot safety techniques.

本项目基于老年/残疾人士的生理和心理特征，利用点云、图像和声音多模态信息，研究智能轮椅室外三维环境认知和安全预警技术，辅助老年/残疾人士认知室外环境，准确预警危险状态，保障舒适安全出行。研制面向智能轮椅的小型化三维激光彩色扫描测距系统，获取三维彩色点云数据，建立室外场景三维彩色几何模型；构造主成分特征和颜色特征，设计基于点云和图像多模态特征的智能轮椅室外场景三维彩色点云多层次分割体系；利用多模态优势和无类标签信息，提出基于点云和图像多模态协同学习的智能轮椅室外场景三维彩色点云分类方法，构建智能轮椅室外场景语义描述；研制智能轮椅前端仿生触碰声音传感系统，提出基于图像和声音多模态协同学习的智能轮椅行进路面材质识别方法；构建智能轮椅碰撞失稳预警模型，建立智能轮椅安全预警机制。本项目的研究不仅具有重要的社会意义和广阔的市场前景，而且对丰富类人感知认知技术和服务机器人安全技术也具有较强的理论意义。

本项目基于老年/残疾人士的生理和心理特征，利用点云、图像和声音多模态信息，研究智能轮椅室外三维环境认知和安全预警技术，辅助老年/残疾人士认知室外环境，准确预警危险状态，保障舒适安全出行。本项目研制了智能轮椅，开发了三维激光彩色扫描测距系统，提出了基于栅格标定器具的激光扫描仪与相机标定方法，获取了三维彩色点云数据，实现了智能轮椅室外场景模型重建；计算了三维点云几何形状特征，构建了主成分特征球，提出了基于主成分特征球的三维彩色点云分割方法，实现了智能轮椅室外场景地面提取和区域分割；利用多模态优势，提出了基于点云和图像多模态协同学习的三维彩色点云分类方法和面向三维点云分类的轻量化注意力模块，提升了智能轮椅室外场景物体分类的精度和实时性；研制了智能轮椅触碰声音传感系统，提出了基于声音和图像多模态协同学习的路面材质识别方法，实现了智能轮椅行进路面材质识别；基于多模态信息，设计了多目标物体跟踪体系，提出了基于离散导数的三维点云几何特性估算方法，分析了智能轮椅失稳原理，构建了智能轮椅碰撞失稳评价机制。本项目在《IEEE Trans. Ind. Electron.》、《IEEE Trans. Instrum. Meas.》等期刊会议发表学术论文13篇，其中SCI论文7篇，EI论文10篇；撰写教材一部，已交稿高等教育出版社；授权中国发明专利8项，申请中国发明专利10项、国际发明专利1项，发明专利成果转化50万元；授权软件著作权3项，获得辽宁省自然科学学术成果二等奖1项；培养研究生19人，其中博士生3人，硕士生16人。本项目的研究具有较好的推广应用前景，对丰富感知认知技术和服务机器人技术也具有较强的科学意义。