时空约束的协同视觉显著性检测与目标跟踪

Video saliency detection and object tracking are usually considered as two independent tasks in computer vision, or the former one is applied to the latter one in a unidirectional form. Although the saliency performance has been improved these years by using deep learning features, the state-of-the-art saliency models are still difficult to extract salient objects from complex background regions in a dynamic video, which leads the saliency-based object tracking models suffering from the limited robustness. This project proposes a unified framework, with the purpose of achieving the robust video saliency detection and object tracking simultaneously. First, a novel motion energy filtering model is proposed to predict the location of salient objects and to suppress the surrounding background regions. Through evaluating the generated motion energy map, some reliable object and background regions are extracted to define a saliency detection model with spatiotemporal constraints which is able to achieve a global saliency optimization. Then, we propose a novel object tracking model with a saliency-guided regularization constant to localize the salient objects. Interactive, the tracking result and the saliency information serve as the semantic prior to further optimize the motion filtering model as well as the saliency detection model to generate a higher-quality saliency map, which is used as a new constraint to boost the performance of object tracking. In this way, the video saliency detection and object tracking models work interactively and iteratively which leads to the optimal results. This project will facilitate to boost the performance of relevant applications in computer vision as well.

视频显著性检测与目标跟踪通常作为两个独立的问题，或者把显著性作为先验知识单向地应用于目标跟踪。尽管近年来深度学习的引入提升了显著性检测的性能，然而面对复杂的动态场景，现有的显著性模型尚不能完整地凸显显著目标和有效抑制复杂背景。本项目提出协同视频显著性检测与目标跟踪的统一模型框架，旨在同时实现稳健的视频显著检测与目标跟踪。首先提出一个高阶语义引导的混合运动能量滤波模型，试图有效估计目标运动能量并滤除无关背景区域。根据运动能量分布，提出基于多维特征聚类的可靠区域检测算法提取可靠的显著目标与背景区域，进而以此为基础建立时空约束的显著性检测优化模型。然后提出了基于显著性空间正则化约束的目标跟踪模型得到目标的定位框。该定位框与显著图一同作为高阶语义进一步优化混合运动能量滤波模型和显著性检测优化模型，从而衍生出更优显著图进一步优化目标跟踪模型。以此协同促进机制，达到显著性检测与目标跟踪的性能最优化。

视频显著性检测和目标跟踪是计算机视觉领域的两个基本问题，通常作为两个独立问题来研究，或者仅是把显著性作为先验知识单向地应用于目标跟踪。在复杂动态场景下，现有的模型无法完整凸显目标对象的边缘信息和抑制复杂背景区域，难以生成质量较好的显著性目标检测结果。针对上述问题，本项目提出协同视频显著性检测与目标跟踪的统一模型框架，旨在实现稳健的视频显著性目标检测与跟踪。首先，提出了一种基于时空约束优化的视频显著性目标检测模型，验证了联合时空信息对显著目标运动估计的可行性。为了进一步提升视频显著对象的检测性能，在联合时空信息思想的基础上，提出弱监督的时空级联神经网络模型用于视频显著性目标检测，并通过注意力机制约束完成显著性映射的细化，有效提升了对缺乏标注的动态视频场景下的显著性目标检测性能。此外，提出了基于虚拟边界引导的视频显著性目标检测模型，解决了显著性目标截断问题。最后，提出多样性正则化等泛化性优化方法，大大提高了视频显著性目标检测跟踪模型的泛化性。.本项目的研究成果对于推动显著性模型及其相关应用的发展具有重要意义。截止2021年底，已发表学术论文26篇，其中，SCI检索的国际学术期刊论文20篇（包括TIP，TCSVT和Neurocomputing），国际学术会议6篇（包括ICCV论文和NeurIPS论文2篇）；申请发明专利18项；培养博士后4名、博士生3名、研究生11名。