基于语义多边图的多物体图像类别发现及其在图像检索中的应用

For real images, the concurrence of multiple objects without their accurate category labels renders the image understanding extremely challenging. Mutual interference among multiple objects within a single image will inevitably introduce misleading information, weaken the representativeness for each of them and consequently limit the final accuracy. A common remedy is to utilize external object category labels to analyze the semantic content of images and help better understand the images. However, such semantic labels are quite rare and unreliable in real images and difficult to be automatically obtained. Under these realistic conditions, a method for simultaneously discriminatively representing multi-object images and accurately revealing semantic relationship of images without requiring external labels becomes extremely demanding in various practical applications, such as image retrieval..In this project, we target to build an integral framework for elegantly and effectively partitioning the involved multi-object images into multiple specific category groups in an unsupervised and automatic manner, which is called unsupervised category discovery of multi-object images. Then we apply it to image retrieval to semantically analyze images and provide superior performance over appearance-based methods..To this end, we first propose one well-designed image representation based on correlated sparse coding, where the representativeness of all characteristic objects is elaborately enhanced through more accurately segmenting out specific subspace for individual object. In practice, both the embedded dictionary learning and coding process will be optimized toward this target. After obtaining image representations, we build semantic multi-edge graph of reference images through auto-grouped sparse representation to explicitly express the semantic relationship between images, where each type of edge represents a certain category without an explicit label. To obtain the final category groups of images, we plan to employ the dense sub-graph discovery tailored for the constructed multi-edge graph, where each discovered sub-graph induces one category group comprising all relevant images. Finally, we apply the proposed category discovery method in semantic image retrieval. Taking into account the adverse effect and interference among multiple objects, certain ranking metric based on partial matching will be constructed to more effectively retrieve images containing interested object while appearance of the image may be quite different to the query..In summary, our works focus on tackling the mutual interference of multiple objects within individual image, from the perspectives of generating more accurate and robust image representation and automatically discovering categories. Then we particularly perform the semantic analysis to improve the performance of image retrieval. Both of these two problems are evidently fundamental in computer vision and multimedia.

现实图像中多个物体间的遮挡和相互干扰会削弱图像表示对物体的描述能力，同时图像的多语义标签难以可靠地获取，这些会显著地影响图像理解任务的性能。因此，根据图像内容生成精确的图像表示并准确揭示它们的语义关系变得尤为重要。本项目的目标是针对现实图像的多物体特性，实现基于内容的自动语义类别发现，并将其应用到图像检索中以引入语义分析来提高检索精度。为此，首先提出了基于相关稀疏编码的图像表示方法，通过子空间约束来增强其对多物体的表示能力。然后引入语义多边图来描述图像间的多种语义关系，并提出了基于自动分组稀疏表示的多边图建立方法，以及基于边图聚类和稠密子图发现的图像分组方法，从而将图像自动划分为不同语义类别组。最后，基于建立的类别发现方法，提出了综合类别语义和局部匹配相似度的语义图像检索方法。通过上述内容的研究，本项目实现了多物体图像的类别发现和语义图像检索，从而为图像理解基本问题提供了一种新的解决途径。

现代图像分析中，待处理图像通常比较复杂（如：包含多个物体，具有杂乱背景，存在干扰噪声等），如何实现这类复杂图像的精确语义理解是视觉分析的基本问题。本项目从三个层面对这一问题进行了研究，包括图像表示、图像内容语义分析和领域应用，目标是更精确地分析出语义类别，并在实际问题中进行应用。.在图像表示方面，本项目开展了多项coding-pooling框架下的研究，重点考虑了影响图像表示质量的三个负面因素：信息丢失、杂乱背景和干扰噪声。具体地，针对高区分力信息丢失，提出了一种基于image-2-class距离的增强图像表示方法LDC（Linear Distance Coding），该方法除传统编码信息外，还捕获了特征编码的残差信息；针对杂乱背景，提出了一种基于块综合选择的方法 HPS（Holistic Superpixel Selection），该方法通过直接选择图像超像素来消除杂乱背景的干扰，而不是显式地分割或检测出物体；针对干扰噪声，提出了一种基于协作方式的线性编码方法 CLC （Collaborative Linear Coding），该方法将噪点消除操作嵌入到编码过程中，通过不同局部特征的编码协作来抑制噪声特征的响应幅度。以上成果以论文形式发表，在多个标准数据集上的结果显示，它们的分类精度能提高2-9个百分点。.在图像内容语义分析方面，本项目重点研究了基于比例信息的多类别分析问题和显著性物体检测问题。具体地，针对大数据情况下图像类别标签难以精确获取的问题，提出了一种仅利用类别比例信息的多类别分类方法，该方法利用单个图像在类别标签上的稀疏性，通过比例信息约束的优化模型可实现其类别判断；针对显著性物体的完整性和一致性问题，提出了一种基于显著性偏置和传播的显著物体检测方法，可有效地检测出显著物体的各部分。以上成果以论文形式发表。.在应用研究方面，研究了具有粗细粒度联合特性的果蔬识别问题。参照《中国蔬菜栽培学》和《果蔬栽培学》从食用角度构建了一个果蔬图像数据集，包括了常用的199个蔬菜类别和91个水果类别，共15万张以上的清晰图像。.综上，本项目围绕图像语义类型分析，从三个层面开展了相关研究并取得了一些阶段性成果。然而，相较于强大的深度神经网络模型，其性能还有很大差距。在后续工作中，将在深度学习框架下开展更加深入的研究，包括将当前成果的核心思想迁移到DNN模型中。