弱监督学习框架下大规模图像语义理解关键技术研究

The aim of our research topic is to accomplish weakly-supervised large-scale image semantic understanding. In order to fully harness social images under weakly-supervised framework, the main contributions of the proposed research topic are as follows: (1) Cost-sensitive label ranking based single image saliency detection algorithm is firstly studied, which casts image saliency detection as a weakly-supervised ordinal regression problem; In addition, by analyzing the fact the multiple images co-saliency detection can also be formulated as a multiple graph matching issue, a multiple graph matching based co-saliency detection algorithm will be discussed. (2) We aim to propose a weakly-supervised multi-label learning based image annotation algorithm which takes the issues of label noise, label semantic correlations and insufficient training samples into consideration simultaneously. (3) We study how to incorporate both self-paced learning and curriculum learning into partial label learning, a special case of weakly-supervised learning manner. By introducing the self-paced curriculum learning regularization, the label discrimination ability of partial label learning could be improved significantly. (4) We aim to propose a matrix completion based social image tag completion algorithm, which aims to automatically fill in the missing tags as well as correct noisy tags for given images. We present the image-tag relation by a tag matrix, and search for the optimal tag matrix consistent with both the visual similarity and pairwise ranking information between observed tags. (5) We study the image captioning problem by introducing external source, as well as combining both bottom-up (local) and top-down (global) attention mechanism to improve the captioning performance and describe novel objects outside of image captioning datasets.

本项目主要研究弱监督学习框架下的大规模图像语义理解，克服互联网图像的噪声标记影响，研究鲁棒的图像理解计算模型。研究内容包括：(1)研究基于代价敏感标记排序的单图显著性检测算法和基于多图匹配的多图协同显著性检测算法；(2)研究基于弱监督多标记学习的图像标注算法，考虑训练样本标记噪声、标记语义相关性挖掘、以及训练样本规模不足等因素，实现鲁棒的图像标注算法；(3)研究融合自步课程学习机制的偏标记学习算法，通过融合课程学习与自步学习的优势，改进偏标记学习算法的学习策略以提升标记消歧性能；(4)研究基于矩阵填充理论的社群图像标签修正算法，通过矩阵低秩正则项约束及标签间排序关系，使得图像-标签关系矩阵能够有效表征标签集合在给定图像上的语义排序关系；(5)研究融合局部自底向上和全局自顶向下两种注意机制的图像描述生成算法，并结合辅助信息实现未知类别对象描述，在弱监督条件下生成精准的图像描述语句。

本项目主要研究弱监督学习框架下的大规模图像语义理解，克服互联网图像的噪声标记影响，研究鲁棒的图像理解计算模型。项目负责人围绕着本项目的研究内容，展开了一系列深入的研究工作。首先在弱监督多标记学习算法方面，围绕着输出空间的标记歧义性问题，重点研究了偏标记学习(Partial Label Learning)、偏多标记学习(Partial Multi-Label Learning)等两类特定的弱监督多标记学习算法；在此基础上，进一步深入挖掘了输入空间的多义性问题，围绕着多视图学习(Multi-View Learning)展开研究，并提出了一系列原创的多视图聚类(Multi-View Clustering)算法、以及弱监督条件下的多视图多标记分类 (Multi-View Multi-Label Learning)算法；最后，针对图像语义理解中的应用问题，提出了弱监督多标记学习条件下的图像分类算法、以及图像显著性检测算法、以图生文等跨模态图像语义分析等。在本项目的资助下，项目负责人冯松鹤以通讯作者发表及录用学术论文共计31篇，其中IEEE/ACM Trans系列会刊论文12篇，包括IEEE Trans. on Knowledge and Data Engineering 1篇，IEEE Trans. on Multimedia 2篇，IEEE Trans. on Cybernetics 2篇，ACM Trans. on Knowledge Discovery from Data 4篇，ACM Trans. on Intelligent Systems and Technology 3篇, 其他主流SCI期刊论文10篇；发表CCF A类会议论文6篇，CCF B类会议论文2篇，其中AAAI 2篇，IJCAI 3篇，ACM SIGKDD 1篇，ECML-PKDD 2篇。累计申请国家发明专利4项。在人才培养方面，项目负责人冯松鹤指导研究生获得博士学位2人，获得硕士学位9人。其中1人获得校级优秀博士论文，3人获得校级优秀硕士论文。.