互联网有限监督信息下的通用图像语义理解机制研究

With the popularity of social media and mobile internet applications, there is an explosive growth of web images. The social characteristics and the increased scalability turn out a great challenge in the image semantic understanding. Though the user's comments and tagging can be well exploited to provide more semantic cues for image semantic analysis, the annotations of these data contain a lot of noisy tags and are always weakly tagging. Thus, the supervision information available is limited due to the huge output space. Furthermore, the negative or testing examples come from an infinite semantic space and we have no clue about the semantic these examples include. In this proposal, we target to develop a framework of ad hoc web image semantic understanding with limited supervision. Based on the recent development and research focuses in multimedia, computer vision, machine learning, and natural language processing, four key issues are explored in this proposal: large-scale supervised dictionary learning with semantic taxonomy, semi-supervised heterogeneous domain adaptation, structural prediction and description generation of image semantic, efficient algorithm and its consistency analysis. Based on the research results of this proposal and related technologies, we will release a portal of image retrieval with semantic ontology. The output technologies and demonstration of this proposal will mainly contribute to the real-world applications of image semantic understanding in web multimedia search, regulation, and services etc.

社会媒体和移动互联应用的发展使得互联网图像数据海量涌现，社会属性和不断增长的规模给图像语义理解带来了巨大挑战。尽管用户评论和标签为图像语义分析提供了更多的语义线索，但这些语义标注信息往往是有噪音和弱标记的。因此，可用的监督信息相对于巨大的语义输出空间是非常有限的。同时，由于负例或测试图像样本在理论上存在的无限语义空间，我们无法有效获得其语义空间的先验信息。本课题拟建立互联网有限监督信息下的通用图像语义理解框架，在结合多媒体、计算机视觉、机器学习和自然语言处理等交叉领域的最新进展和热点研究的基础上，主要研究：语义层次监督信息下的大规模字典学习、半监督异构的跨域学习、图像结构化语义预测和语义描述生成、以及算法高效求解和一致性分析等内容。通过集成相关研究成果与技术，发布基于本体语义的图像检索系统，为图像语义理解在互联网媒体搜索、监管与服务等领域的实际应用提供技术支撑和平台示范。

在互联网图像语义理解中，尽管用户评论和标签为图像语义分析提供了更多的语义线索，但这些语义标注信息往往是有噪音和弱标记的。本项目主要研究了：语义层次监督信息下的大规模字典学习、半监督异构的跨域学习、图像结构化语义预测和语义描述生成、以及算法高效求解和一致性分析等内容。并构建了图像语义理解数据集和基图像检索原型系统，验证了算法和框架的有效性。项目成果在国内外重要期刊/会议发表论文43篇，包括IEEE Trans.论文5篇；CCF-A类会议论文8篇，CCF-B类会议论文6篇；相关技术成果申请国家发明专利5项；相关算法在MSR Video-to-Language等国内外评测中取得优异成绩。为图像语义理解在互联网媒体搜索、监管与服务等领域的实际应用提供技术支撑和平台示范。