基于神经网络框架的RGB图与深度图相结合的行人再识别若干关键问题研究

There are variations in the same pedestrian across different cameras, including illumination, viewpoints and resolution, etc, which makes person re-identification challenging. The depth images contain the depth information, which is invariable to the variance of the illumination and color, etc. Combining the depth image with the RGB image to accomplish the person re-identification task, can achieve the functional complementarity of these two images. At present, only a few person re-identification tasks are implemented by combining these two images, and most of them extract the handcrafted features and concatenate these features, which is not suitable to extract abstract semantic depth information of pedestrian. Furthermore, the simple feature concatenation can not adapt to the two image features of different magnitude orders. This project intends to adopt a combination of the depth image and the RGB image to realize person re-identification through the deep network framework. On the one hand, the twin-flow network structure is employed to extract and merge the distinctive RGB and depth image features respectively. Moreover, the salience method is applied to perform the person re-identification and improve the accuracy of it. On the other hand, based on RGB images, the depth images are estimated through prior knowledge of neural network learning. To expand the datasets for solving the lack of depth images and deep network training datum problem, the generative adversarial network is exploited to generate pedestrian videos. The outcome of this project will be of great significance to the development of artificial intelligence and national public safety.

相同行人在不同摄像机下存在光照、角度、分辨率等差异，致使行人再识别任务颇具挑战性。深度图中的深度信息对光线、颜色等变化具有不变性，将其与RGB图像结合进行行人再识别可以实现二者功能上的互补。目前仅有少数行人再识别任务通过两种图像结合的方式实现，且大多提取手工设计的特征并进行简单的串联，不适合行人深度抽象语义信息的提取，简单的特征串联也无法适应两种图像特征数量级不同的情况。本项目拟采用深度图与RGB图像结合的方式，通过深度网络框架实现行人再识别。一方面，采用双流网络结构融合其分别提取的具有区分性的RGB图及深度图特征，并引入显著性方法来完成行人再识别任务，提高行人再识别准确率。另一方面，拟基于RGB图像，利用神经网络学习的先验知识进行深度图估计；采用GAN网络生成行人视频的方式扩充数据集；解决深度图缺乏和深度网络训练数据不足的问题。项目成果对于促进人工智能学科发展和国家公共安全具有重要意义。

相同行人在不同摄像机下存在光照、角度、分辨率等差异，致使行人再识别任务颇具挑战性。深度图中的深度信息及红外图中的热量分布，对光线、颜色等变化具有不变性。将深度（红外）模态图像与RGB模型结合进行行人再识别，可以实现不同模态功能上的互补。目前部分方法在不同的模态间实现行人的识别，且大多提取手工设计的特征并进行简单的串联，不适合行人深度抽象语义信息的提取，简单的特征串联也无法适应两种图像特征数量级不同的情况。本项目通过深度网络框架实现不同模态间的行人再识别。一方面，采用双流网络结构融合其分别提取的具有区分性的RGB图及深度图特征，并引入显著性方法来完成行人再识别任务，提高行人再识别准确率。另一方面，基于RGB图像，利用神经网络学习的先验知识进行深度图估计；采用GAN网络生成行人视频的方式扩冲数据集，解决深度图缺乏和深度网络训练数据不足的问题。项目成果包括发表的论文17篇，授权的专利3项，软著3项。依托项目培养了研究生共计15名。项目成果对于国家公共安全具有重要意义。