基于视觉注意力与手势动作特征建模的自然交互界面优化方法研究

There are usability deficiencies in natural user interfaces of virtual reality (VR) applications, including low efficiency in tracking targets, poor accuracy in gestural operation and obvious arm fatigue. A major reason for these deficiencies is that user’s visual attention and gestural operation features had seldom been concerned or systematically investigated in this field, and therefore natural human-computer interaction interfaces in VR are often designed randomly without applicable guidelines or standards. The study of user’s visual attention and gestural operation features is complicated and difficult as it involves interdisciplinary knowledge in neurophysiology and kinesiology, which results in obvious gap in current research on natural human-computer interaction and related user interfaces. To fill the gap, this program plans to conduct a deep research on user’s sensory and motor functions, and based on this concludes user interface design guidelines as well as technical methods for building optimized user interfaces in VR environment. There are 3 main research contents in this program: (1) evaluates user’s visual tracking behavior and attention features in VR spaces, through a target tracking experiment and a data visualization analysis method of eye movement trajectory; (2) evaluates free hand interaction efficiency and accuracy in VR spaces, through a free-hand target selection experiment and an updated assessment model of Fitts’ Law; and (3) evaluates arm fatigue level in free hand interaction tasks, through a static mechanical analysis method and a surface EMG signature analysis method. Based on expected findings from above research, this program will then conclude natural user interface design guidelines in VR environment, and build empirical models of visual attention and gestural operation features as well as assessment model of arm fatigue level. Given these models, this program will finally propose an implement method of optimized user interface in VR, i.e., intelligent perceptual user interface (IPUI), and have its optimization effects verified.

虚拟现实自然交互界面中存在“目标搜寻效率低、手势交互精确性和舒适性差”等可用性问题。上述问题主要是因为在界面设计实践中缺少对用户视觉注意力与手势动作特征的系统研究，而以上研究涉及神经生理学、人体运动学等多学科知识，是传统人机界面研究中存在较多盲区的地方。为了弥补上述不足，本项目将从人类感知运动功能入手，重点开展3方面研究：（1）通过目标追踪实验和眼动数据可视化分析方法对用户的视觉搜索行为与注意力特征进行评估；（2）通过目标抓取任务以及重建的Fitts’Law评估模型对手势交互效率和精确性结果进行评估；以及（3）通过静态力学分析和阵列式手臂表面肌电信号（sEMG）特征分析方法对不同手臂姿势下的疲劳度进行评估。本项目将基于以上结果，形成虚拟现实自然交互界面设计规范，并构建视觉注意力与手势动作特征经验模型以及手势疲劳度评估模型，最后运用以上模型提出一种智能感知用户界面生成方法并验证其优化效果。

以上肢活动和自然手势操作为基础的人机交互技术是当前“自然人机界面”形态中应用最为普遍的一种方式，也是以“元宇宙”应用生态为代表的数字人机融合发展趋势下发挥重要基础性作用的一项关键技术。本研究从人体运动学与上肢生物力学结构特征出发，结合视觉感知与上肢运动行为分析以及人因工程与工效学实验测评方法对虚拟现实自然人机交互应用场景下的信息感知与手势操作特征以及一般性手势交互任务完成绩效展开研究，分别从（1）注意力特征与视觉感知行为特征对目标追踪效率影响机制、（2）上肢关节运动特征与自然手势操作能力对手势交互任务影响机制以及特征模型表示、（3）上肢关节运动特征与上肢操作疲劳度感知关联性以及疲劳度感知模型建构、（4）基于视觉感知能力与上肢关节运动特征的手势交互增强技术等4个方面具体展开，通过实证研究与实验分析方法证明了（a）视觉感知信息效率评估中存在“最佳感知距离”和界面显示区域“最佳视场角”（optimal FOV）；（b）上肢运动操作在虚拟现实交互空间中的“非对称性”特征（asymmetry）和“惯用手优势”及其表现在不同移动方向与角度上的手势交互效率和精确性差异化结果；并验证了（c）上肢操作在不同关节运动姿势和弯曲角度下的疲劳度感知差异；也探索了（d）在沉浸式自然人机交互空间中视觉感知行为与自然手势操作行为之间相互独立又强关联的工效学特征。研究成果为揭示自然手势交互技术应用中普遍存在的可用性问题提供了人因工效分析视角，也为虚拟现实、增强现实等应用场景下自然手势交互系统可用性质量优化和工效学品质提升提供人机界面设计规范和交互技术优化策略。本研究还以“手势交互界面自适应技术”为例，介绍并展示了自然手势交互技术增强方法，为开发面向未来“元宇宙”等应用场景的个性化智能人机交互技术提供发展思路。