复杂环境下Web内容质量评价的统计学习方法研究

Web content quality assessment is crucial to many Web content processing applications, such as Web search, Web archive and Internet directory. In recent years, web spams, phishes, false advertisements and web gunmans have flooded in the Internet with rapidly increasing levels of variety and disguise. These challenges make it more and more difficult to effectively assess Web content quality. This project proposes to systemically investigate several core techniques of Web content quality assessment, including feature extension, new assessment methods and semi-supervised learning strategies. The key techniques we are going to explore include: (1) to expand the existing feature space by extracting a series of Web spam, domain name resolution and temporal features, with expectation to find a proper feature space well reflecting the Web content quality; (2) to adapt the learning to rank algorithms to Web content quality assessment, and explore new learning algorithms via directly optimizing linear NDCG measure; (3) to study multi-view semi-supervised Web content quality assessment strategy which is able to boost the assessment abilities of multiple learning algorithms, the topologic dependencies of different quality nodes, as well as the prediction performances of different link analysis algorithms; (4) to explore crowdsourcing based Web content quality dataset construction in order to avoid expensive manual annotation. This project will be conducted on the basis of our previous research. Most technology difficulties have been comprehensively evaluated. The research accomplishments are expected to provide solid theoretical and technical supports to various Web content processing applications.

Web内容质量评价是互联网搜索、网络内容归档和互联网公开目录等的核心算法。近年来，Web作弊、网络钓鱼、虚假广告和网络水军等充斥网络，且表现出越来越高的隐蔽性和多变性，给Web内容质量评价造成极大困难。本项目拟对Web内容质量评价各阶段关键技术进行探索：1）从Web作弊、域名解析和网站成长模型三个角度拓展现有特征空间，以提取更能反映Web内容质量的特征；2）研究将排序学习方法用于内容质量评价的策略，在此基础上，通过直接优化质量评价的度量指标以探索新的评价算法；3）研究可拓展的多视图半监督Web内容质量评价策略，拟有效集成学习算法的评价能力、不同质量节点的拓扑依赖和链接分析算法的预测能力；4）针对人工标注成本高的事实，探索基于众包的Web内容质量数据集构建方法。本项目在前期良好的研究基础上展开，针对研究中的难点设计可行技术方案。本项目研究成果将为多种互联网内容处理应用服务提供理论和方法支撑。

Web内容质量评价在多种互联网内容处理应用中起到重要作用。围绕复杂环境下的Web内容质量评价，本项目开展了多项关键技术研究，包括：1）提取了多尺度特征，包括：字符串特征、域名注册特征、超链接特征、第三方服务特征、DNS解析特征、Web作弊特征和时序特征。2）研究低质量Web内容（暗链、色情、赌博、钓鱼等滥用信息）的识别技术。3）探索Web内容质量评价新方法，包括融合多个二值pair-wise模型的排序算法和直接优化线性NDCG损失的评价算法。4）开展半监督Web 内容质量评价研究，实现在少量标注样本下的Web内容质量评估。5）探索新的样本标注途径，提出了基于DNS解析日志的Web内容质量数据集自动标注模式。6）开展实体搜索排序研究，所提出的算法在2016年CCF自然语言和智能分析竞赛中获得第一名。本项目研究成果为Web内容理解服务提供了有益的思路和方法。