微创手术中体表投影虚拟透明观察内腔方法研究

In the field of minimally invasive surgeries, body surface projection approach, which helps the surgeon to observe the lumen of the patient, is a research focus. However, the approach is still in the phase of exploration and attempt, and its effect is limited by accuracy, resolution, real time and influence on surgical environment, which still can not meet the requirement of endoscopy. . Focused on the existing problems, we primarily introduce the Kinect body-tracking technique to build the eyes and nose depth data of the surgeon as well as the mathematical model of the surgeon’s view orientation, which is able to offer an accurate and real-time tracking of the surgeon’s view angle without affecting the surgical environment. Furthermore, we introduce the PTAM 3D modeling technique and propose a lumen feature system based on the branch points and segments of the vessel, which achieves an accurate, dense and texture-included 3D model of the narrow and feature-repeated lumen. Finally, combining the surgeon’s view angle and the lumen model, we propose a real-time projecting solution which is applicable for non-quadric body surfaces, multi-projector overlap and the view transformation of the surgeon. The solution is able to display a virtual transparent lumen. Aimed at the approach mentioned above, we design the system and implement the laboratory and operating room experiments to have an evaluation.. The research is a interdisciplinary of computer vision and measurement control. It aims at improving the comprehensive visual effect but not sticking at the accuracy and resolution of a certain part. Simultaneously, the research pays close attention to the development of commercial equipment and techniques which may have a contribution to the system. Relevant techniques can expand to fields like game entertainment, industrial measurement and military training.

微创手术中辅助医生观察患者内腔的体表投影法是该领域的研究热点，但尚处于探索和尝试阶段，其效果受准确度、分辨率、实时性、影响手术环境等限制，仍未满足手术内窥需求。.针对尚存问题，首先引入Kinect肢体追踪技术，构建医生眼、鼻深度数据与视角方位的数学模型，准确、实时追踪医生视角且不影响手术环境；其次引入PTAM三维建模技术，提出基于血管分支点和分支段的内腔特征体系，实现狭小、特征重复内腔的准确、致密、含纹理三维建模；最后综合医生视角和内腔模型，提出适用于非二次曲面体表、多投影机交叠、医生视角变换的实时投影方案，获得真实的虚拟内腔透明效果。针对上述方法，设计组建系统，进行实验室和手术室实验评价。.研究属于计算机视觉和测量控制的学科交叉，着重提高综合视觉效果而不拘泥于某环节的准确度和分辨率，并兼顾先进商业设备和技术的发展对系统实用性的贡献。相关技术可扩展应用到游戏娱乐、工业测量、军事训练等领域。

微创手术中辅助医生观察患者内腔的体表投影法是该领域的研究热点，针对其效果受准确度、分辨率、实时性、影响手术环境等限制，本项目探索医生视角追踪、内腔三维建模、投影畸变校正三方面改进，促进其更好地应用于微创手术领域。构建体表投影虚拟透明观察内腔系统，总体投影位置误差xyz方向综合约3mm，内腔三维建模、体表三维测量等预标定时间约3分钟。.首先，选择瞳孔、鼻尖的深度数据作为关键点，建立医生视角数学模型，进而引入卡尔曼滤波器预测医生视角并修正追踪数据。头部追踪位置误差约3mm，角度误差约3°，追踪帧频约10Hz。相关技术是AR、计算机视觉、测量控制领域的研究热点，也是将方位追踪技术更好地应用于游戏娱乐、医学学习、军事训练、工业测量等领域的关键。.其次，建立ORB-SLAM2内腔标定和测量条件，构建血管分支点和分支段内腔特征体系，针对狭小、特征单一、包含镜面反射区的内腔空间建立三维模型。利用vivodataset和Hamlyn dataset医学影像数据库比对配准准确度、致密度。构建含纹理可旋转内腔三维模型，三维重建RMSD约2.2mm。相关技术同样适用于图像拼接、图像检索、视频追踪等应用领域。.再次，为了得到更真实的虚拟腹腔透明效果，综合医生视角和内腔三维模型，生成患者体表投影图像序列。融合区域未出现错位现象，边缘细节完整且连续，融合视觉效果好。观察视角实时变换的投影准确度、分辨率、速度研究，促进其在AR相关的影视、展览、医学等领域更好地应用。.最后，根据相关技术的最新进展，拓展研究了深度学习视觉测量特征点检测与匹配、胸腹表面呼吸运动的数学描述和规律等问题，并应用于医学图像分类、医学图像检索、人脸识别、微血管图像分割、无人机图像去雾等领域。