面向网络图像检索的弱监督多模态跨域机器学习方法研究

Semantic gap is a main challenge to image retrieval theoretical research and applications during the past two decades. In the Internet era, the explosively-growing web images expose some new characteristics, including cross-view, cross-modality and cross-domain, which provides some novel ideas to address the semantic gap problem. By taking the 'triple-cross' characteristics of large scale web images into consideration, this project conducts a systematic study on the weakly-supervised multi-modality cross-domain machine learning theory and methods, with the goal of narrowing down the semantic gap. Our research mainly concentrates on the cross-view refinement of weak supervision signals, the cross-modality analysis of image visual content, and the cross-domain transfer of images' social structural feature, in order to improve the usability of crowdsourcing data, utilize the complementary properties of different visual features, and mine the mainstream intention of web users, which will be beneficial to enhancing search engine's visual sensing capability and users' experience. To our best knowledge, there has been little study on exploiting the 'triple-cross' characteristics of large scale web images, and not to mention any systematic theory and methodology. The implementation of this project will further enrich the research of image retrieval field, promote the industrialization progress of image retrieval, and be expected to yield some innovative outcomes.

语义鸿沟问题是过去二十年间制约图像检索理论研究与实践应用的主要瓶颈。在互联网时代，爆炸式增长的图像数据日益表现出跨视图、跨模态和跨域三个新特性，进而为研究语义鸿沟问题提供了新思路。本项目以大规模网络图像数据为研究对象，围绕其“三跨”特性，系统研究弱监督多模态跨域机器学习理论与方法，以期有效缩小语义鸿沟；重点开展弱监督信号的跨视图提炼、图像视觉内容的跨模态分析、图像社会化结构特征的跨域迁移三个方面的研究，旨在有效提高众包数据的可用性、充分利用多模态视觉特征之间的互补性、深度挖掘网络用户的主流性信息需求，从而提升搜索引擎的视觉感知能力并增强用户体验。截至目前，针对网络图像“三跨”特性的综合性研究国内外鲜有报道，更没有形成系统性的理论和方法体系。本项目的实施将进一步丰富和完善图像检索研究领域的新内容，推动图像检索产业化进程，有望取得一些创新性成果。

针对网络图像的“三跨”特性（跨视图、跨模态和跨域），本项目以大规模网络图像数据为研究对象，系统研究了弱监督多模态跨域机器学习理论与方法，以期有效缩小语义鸿沟。研究重点主要聚焦于：1）弱监督信号的跨视图提炼、2）图像视觉内容的跨模态分析、3）图像社会化结构特征的跨域迁移三个方面，旨在有效提高众包数据的可用性、充分利用多模态视觉特征之间的互补性、深度挖掘网络用户的主流性信息需求，从而提升搜索引擎的视觉感知能力并增强用户体验。经项目组成员通力合作，共计发表学术论文25篇，其中SCI论文14篇、国际顶级期刊2篇（IEEE TPAMI）、顶级会议2篇（AAAI）；申请国家发明专利5项，其中1项授权并实现成果转化；培养青年教师3人（晋升为副教授）、硕士研究生7人（毕业），实现了科学研究与人才培养双丰收的预期研究目标。