融合注意机制的视觉计算方法研究

Selective visual attention is a very important information processing mechanism of human visual system. The mechanism exploration and computational modelling of visual attention has been the important research problems in the fields of psychology, computational neuroscience and computer science. Current research focuses on the static characteristics of visual attention: visual saliency, and ignores to model the dynamic characteristics of visual attention: saccadic scanpath. It is also rare to investigate the dynamic characteristics of visual attention in the advanced computer vision tasks, e.g. object detection and recognition. The aim of this project is to model the dynamic characteristics of visual attention, integrate the dynamic eye movements into visual computing tasks, and propose new visual computing methods. With the guidance of visual attention, we hope to break through the bottlenecks of current visual computing system. Given the advantages of deep neural network and their recent series of successful applications, all the research work of the project will be based on deep neural networks. Through the implementation of this project, on the one hand, we hope to improve the study of the dynamic characteristics of visual attention; on the other hand, integrating the active vision mechanism of visual attention into the existing visual computing systems is expected to improve object detection and recognition in terms of efficiency and precision; Finally, we will propose series of new deep neural networks to effectively handle visual data. This research project is highly theoretical and of highly practicality.

选择性视觉注意是人类视觉系统中一个非常重要的信息处理机制，然而目前视觉注意计算模型主要集中在静态显著性建模，忽略了对视觉注意更为重要的动态眼动特性（扫视路径）的建模，而且当今计算机视觉领域也很少研究视觉注意动态特性在目标检测与识别等高级视觉任务中的作用。本项目的目标是对视觉注意的动态眼动特性进行建模，并在目标检测与识别等视觉任务中融合动态眼动模型，从而提出新型视觉计算方法，期待在注意机制的指导下突破现有视觉计算系统的瓶颈。鉴于深度神经网络自身优势和最近一系列成功应用，本项目的所有研究工作都会在深度神经网络的基础上展开。通过本项目的实施，一方面完善了视觉注意的动态特性研究，另一方面在现有视觉计算系统中融合注意机制有望提高目标检测和识别等任务的精度和效率，最后还将提出更有利于处理视觉信息的一系列新型深度神经网络。本研究项目具有很强的理论性和实用性。

沿着视觉认知计算这一前沿交叉方向，在类人智能视觉系统为目标的指引下，围绕视觉认知计算建模、结构关系计算建模、视觉语言计算建模这三个问题开展了理论和关键技术研究。本项目在四年执行期内共发表论文16篇，国际期刊论文3篇（包括2篇TPAMI，1篇PR），国际会议论文13篇（其中包括本领域顶级国际会议文章9篇，5篇CVPR，2篇ACM MM，AAAI、ECCV各1篇），申请发明专利2项，培养博士生7名、硕士生1名。在视觉认知计算建模方面，提出了多模态记忆和层次化记忆模型，实现了视觉语言的有效存储与检索；在结构关系计算建模方面，提出了递归卷积网络和递归图卷积网络，解决了视觉序列的时空结构化分析难题；在视觉语义鸿沟方面，构建了视觉语言多粒度注意模型，提出了选择性多模态匹配方法，攻克了跨模态跨领域的视觉语言检索。本项目几项关键技术已经达到当前国际最好水平，相关研究成果应用在国家电网“电力系统智能监控与调度”的相关生产部门。