基于深度学习的协同目标分割与行为理解研究

Human can extract moving object and infer the object behaviors simultaneously in complex scenes. However, it has proven to be a challenging task for computer vision systems. This study pursues a joint solution to the problems of image segmentation and object behavior understanding by combining high-level prior and low-level data. The collaboration between high-level and low-level can effectively enhance the segmentation results of target object and improve the behavior inference simultaneously in low-quality conditions. Our work effectively uses deep learning to model the hierarchical architecture of object behavior and shape priors. The obtained such multiple levels of representation can be exploited to guide variational image segmentation. Shape as a distinctive characteristic of objects, is adopted to link top-down object segmentation with bottom-up behavior inference. By using inference and generation algorithms of multiple deep Boltzmann machines (DBMs), a general variational model and its energy minimization can be built by fusing low-level image segmentation, mid-level object recognition and high-level behavior inference over image sequences. Moreover, the established model can be employed in the resolution of finger-spelling recognition, and the major research issues will be conducted including the representation of prior action, shape-driven variational segmentation, collaborative learning based on multi-DBMs, etc. This study will provide new ideas and approaches for collaboration between low-level image segmentation, mid-level object recognition and high-level behavior understanding.

人类可以在混乱的场景中实现运动目标的分割、识别和行为理解，但这对于计算机视觉系统却是一个严峻的挑战。本项目研究关于高层先验知识指导和底层图像数据驱动相结合，同时进行目标分割和行为理解的方法。该方法能够在实际环境比较恶劣的情况下，实现精确的目标分割和准确的行为推理。主要研究利用深度学习的多层表达能力捕捉目标行为和动作的层次结构特征，结合变分图像分割算法，以目标形状作为上下层连接的纽带，通过多个深度玻尔兹曼机的近似推理和产生机理实现底层、中层和高层的协同合作，建立协同目标分割和行为理解的计算模型，并提出模型的优化及求解方法。同时以手势识别系统中手指拼写单词作为Demo实例，分析研究先验行为的表达、基于形状驱动的变分分割，多个深度玻尔兹曼机的协同学习等关键问题。此研究将为底层序列图像分割、中层目标识别和高层行为理解的协同合作提供新的思路和途径。

本项目研究基于先验知识和底层数据相结合的目标分割与行为识别的基础理论与方法，其主要成果包括：基于深度玻尔兹曼机的协同目标分割与行为识别，基于循环相似性的目标形变配准，基于形状稀疏凸组合的目标分割与识别，基于低秩约束的序列图像协同分割等。此研究重点解决了目标行为表达，多个深度玻尔兹曼机协同学习,目标形变不变性, 稳健分割等问题，同时为底层图像分割、中层目标识别和高层行为理解的协同合作提供新的解决方法。由于研究的对象是低质量图像，噪声等干扰因素往往影响分割与识别算法的效果，现研究成果给图像复原问题也提供了新的研究思路和理论依据，取得了很好的效果。本项目的研究有助于提高计算机视觉系统人机交互的质量和稳健性。