人机共融多模态传感软体操作理论与应用

For the prosthesis soft hand’s fusion problem, including perception, interaction and behavior, this project carries out research on the theory and method of soft manipulation based on human-machine collaboration and multi-modal perception. Then, test and verify the application. Based on the pressure sensing with organic polymer nanometer composite material modification, bend sense with optical fiber and 3D force sensing with micro-vision, the multi-modal soft sensing system is built to realize the perception fusion for the prosthesis soft hand. Research the EEG decoding method based on transfer learning and design the hybrid asynchronous BCI for the interaction fusion. Research the imitation learning method based on the antagonistic generative model for the multi-task to realize the humanoid autonomous manipulation of the prosthesis soft hand. Build the adaptive hierarchical perception fusion model of BCI information and soft hand sensing information by the hidden Markov model and dynamic Bayesian network inference method. Propose the fusion manipulations theory of BCI control and autonomous control. Develop the multi-modal perception prosthesis soft hand and build the experimental platform to validate the feasibility of human-machine collaboration manipulation theory. This research will promote the study of rigid-flexible coupling robot with the bionic feature to break the integrated design of soft structure and driving perception. It has very important scientific significance and application value for the upper limb disability rehabilitation and prosthetics program to improve the development of upmarket robot related industries in China.

面向假肢软体手的传感共融、交互共融和行为共融问题，研究人机共融多模态传感软体操作理论与方法，并进行应用验证。研制纳米面压力传感器、弯曲力传感器和三维力指尖传感器，构建多模态柔体传感系统，实现假肢软体手的多模态信息感知。研究基于迁移学习的脑电解码方法，设计混合异步脑机接口，实现脑机交互共融；研究基于对抗生成模型的多任务模仿学习方法，实现假肢灵巧手的拟人自主操作；采用隐马尔可夫模型与动态贝叶斯网络推理等理论方法，建立脑机接口信息和软体手传感信息的自适应分层共融感知模型，提出脑机接口控制和自主控制的共融操作理论方法。研制多模态传感假肢软体手，建立实验平台，验证人机共融操作理论方法的可行性。研究成果有望推动以仿生为主要特征的刚柔软体耦合机器人研究, 突破软体结构、驱动感知的一体化设计，为上肢残弱人群提供康复训练和义肢方案，推动我国高端机器人相关产业的发展，具有十分重要的科学意义和应用价值。

面向假肢软体手的传感共融、交互共融和行为共融问题，研究人机共融多模态传感软体操作理论与方法，并进行应用验证。在Soft Robotics，Science Robotics，Science Advances，Nature Communications等发表论文84篇，获得了ROBIO最佳学生论文奖、ICRA最佳论文提名奖等多项国际学术会议论文奖。研制了高密度压力触觉传感器阵列、包芯结构的纤维状柔性应变传感器，实现了触觉及手指关节弯曲程度的测量；通过设计微结构和多协同导电网络，提高了压力和应变传感器的敏感特性；基于多模态传感器阵列，实现了主从手交互反馈力感知操作；设计了多模态的基于视觉的触觉传感器并拓展到软体手的感知和抓取操作；针对生物在非结构化环境中的高效作业方式，探究章鱼触手“缠绕-吸附”协同抓取方式，实现仿生章鱼触手机器人样机，提出了基于传感反馈的变曲率弯曲软体机器人建模及控制方法；针对软体机器人自由度多，编程控制困难等问题，基于摩擦纳米发电机原理和液态金属压阻效应设计并研制了适用于软体机器人大变形的柔性双模态智能传感器，并提出了软体机器人非接触示教人机交互新范式；结合模态融合感知技术，开展了五指软体手感知和假肢灵巧手操作交互的研究，实现了脑机接口和增强现实的人机交互。本项目充分结合仿生技术与软体机器人技术，开发的多模态感知柔性手有效提升了上肢残弱人群的操作能力，并通过多个实验平台验证了提出人机共融操作理论与方法的可行性和优越性。相关成果入选了国家“十三五”科技创新成就展。