联合视觉与自然语言的端到端行人再辨识研究

Pedestrian re-identification aims at spotting a person of interest in other cameras, and is required to have cross-modal characteristics in practical application. Based on deep learning theory, this project studies the end-to-end cross-modal pedestrian re-identification by integrating computer vision and natural language, and products some novel theories and methods for the simultaneous pedestrian detection and re-identification under semantic association between image and text. The main research contents are as follows: Firstly, we study new methods to automatically detect pedestrians from image, and to construct the semantic mapping model for the pedestrian images; Secondly, we study new algorithms of word embedding for the description sentence about the pedestrian’s appearance; Thirdly, we study the semantic association model between the pedestrian image and its descriptive sentence; Finally, we study the relevance feedback mechanism and network training strategy. Innovation of the project is as follows: Firstly, the feature learning, pedestrian detection and re-identification are integrated by using the deep neural network; Secondly, cross-modal pedestrian re-identification is achieved based on semantic association between image and text. The research of the project will promote the development of pedestrian identification technology in a more practical direction, and accelerate the exploration of visual and auditory cognitive mechanism. It will improve the computer's processing efficiency and understanding of heterogeneous information, and has important scientific significance and application prospect.

行人再辨识旨在对跨视场中的行人进行身份再认，实际应用要求其具有跨模态性。本项目从联合计算视觉与自然语言出发，以深度学习理论为基础，研究端到端的跨模态行人再辨识，为实现图文语义关联下的行人检测与再辨识一体化提供新理论和新方法。主要研究内容包括：行人图像的自动推荐及其视觉特征语义映射的新方法；行人描述语句的词嵌入新方法；行人图像与其描述语句间的语义关联模型；相关反馈机制与网络训练策略。创新之处：利用深度神经网络实现特征学习、行人检测与再辨识的一体化；利用图文语义关联实现行人再辨识的跨模态化。项目的研究将推动行人再辨识技术朝着更实用化的方向发展，并促进对视听觉认知机理的探索，为提高计算机对异构信息的处理效率和理解能力做贡献，具有重要的科学意义和应用前景。

行人再辨识旨在对跨视场中的行人进行身份再认。在许多情况下，只有关于行人外貌特征的口头描述，即由听觉感知而成的自然语言描述，而以图搜图行人再辨识技术对此无能为力，因此亟需拓展行人再辨识技术的跨模态能力。为此，课题组从联合计算视觉与自然语言出发，以机器学习理论为基础，研究了跨模态的行人再辨识相关理论和方法，主要研究内容包括：行人图像和行人描述语句的语义特征提取与关联方法、跨模态行人再辨识网络的训练和优化策略、行人场景图像的语义分割与自动描述方法、以及行人对象的检测与跟踪方法。取得了以下研究成果：9种行人再辨识方法、6种目标检测方法、4种目标跟踪方法、6种场景图像的自动描述方法和3种场景图像语义分割方法，这些方法的性能指标均达到或超过了其相应领域的最新水平。以上成果被国内外重要学术刊物录用和发表，其中SCI源刊13篇，CCF推荐会议15篇，6项成果获得了国家专利保护。本研究推动了行人再辨识技术朝着更实用化的方向发展，提高了计算机对图文信息的跨域理解能力，具有重要的科学意义和应用前景。依托本项目，培养了人工智能领域人才27人，其中在读博士生3人、获得硕士学位10人。围绕本项目，开展了国际国内学术交流11次，其中作受邀报告2人次、分组报告7人次。