基于神经主题模型的图像分类与标注

Consider that most of image classification and annotation methods based on topic model only apply to discrete bag of words feature, this project is based on neural topic model, introduces continuous convolutional neural network feature, namely Places-CNN, and tries to achieve innovation on image classification, image annotation and simultaneous image classification and annotation. It has a strong focus on the following three science problems: (1) introducing Places-CNN feature in neural topic model, (2) modeling ensemble classification based on neural topic model, (3) making image classification and annotation promote each other in simultaneous image classification and annotation. The main content includes three aspects. Firstly, we introduce Places-CNN feature into neural topic model, study the relationship between image and annotation words, and construct image annotation method. Secondly, based on neural topic model which has introduced Places-CNN feature, we adopt ensemble learning idea to construct image ensemble classification method. Finally, we study how to combine image classification and annotation neural topic model that we have already built, construct a simultaneous image classification and annotation method, and make the two tasks promote each other. The study of this project has important significance for the theoretical research image classification and annotation, as well as improving performance of multimedia retrieval in which image classification and annotation are key techniques.

考虑目前大多基于主题模型的图像分类和标注工作仅适用取值离散的词袋特征，本项目建立在神经主题模型基础上，引入连续的卷积神经网络特征 Places-CNN，力争在图像分类、标注以及同时做图像分类和标注的方法上实现创新。重点解决三个科学问题：（1）神经主题模型中Places-CNN特征的引入；（2）神经主题模型中集成分类的建模；（3）同时做图像分类和标注的神经主题模型中分类和标注的互相促进。主要内容包括：首先，在神经主题模型中引入Places-CNN特征，研究图像和标注词间关系，构建图像标注方法；其次，在引入Places-CNN特征的神经主题模型基础上，结合集成学习思想，构建图像集成分类方法；最后，结合建立的图像分类、标注神经主题模型，构建同时做图像分类和标注的方法，实现分类和标注互相促进。本项目的开展对图像分类和标注的理论研究，以及以图像分类和标注为关键技术的多媒体检索的性能提升具有重要意义。

图像分类和图像标注都是实现图像理解的重要手段。目前，基于主题模型和深层神经网络的图像分类和图像标注工作已取得了丰硕成果，在一定程度上缩短了图像语义鸿沟，但仍然存在一定的挑战。本项目研究图像词袋特征的提取过程、研究深度语义特征的建模方法，提出了基于主题模型和深层神经网络的特征提取方法，从而提高的图像分类和图像标注的性能；研究图像标注的主题模型，针对现有的图像标注方法大多将不同类别的图像置于同一主题空间下进行标注，提出了依据图像类别信息标注图像的方法，进一步提高了对图像局部信息的理解能力；研究实现图像集成分类的深层神经网络的建模，提出了一系列面向少量训练样本的神经网络集成分类方法，提高了图像分类的稳定性；为了进一步提高当前神经网络结构和损失函数的特征学习能力，分别提出了面向判别性特征学习的神经网络结构和损失函数，从而提高图像分类的性能。该项目的研究对图像分类和标注的理论研究、推动多媒体检索技术以及提高机器对真实图像的理解能力都具有重要意义，尤其对少量标注样本的图像理解问题具有一定的借鉴意义。