基于蚁群算法和云模型的领域无关数据清洗

Data quality issues have been restricting our informationization construction seriously, while the data cleaning is a crucial technical means for data quality improvement. At present, Feature-Based Similarity (FBS) methods and Relationship-based Data Cleaning (RelDC) are the two major types of methods for domain-independent data cleaning. FBS has efficiency superiority, however it has some disadvantages such as low similarity function precision, poor performance of feature selection algorithm, and disability for complex data records; RelDC is superior in effectiveness, but it is difficult to be carried out, and it can't switch with FBS properly. In this project, we will propose several new FBS methods by defining the similarity functions of different attribute types and determining their parameters, introducing non-linear classifier and designing the classifying detection process of approximately duplicate records, and constructing a mathematical model of the feature selection and ant colony algorithm to solve the model. Then, we will present RelDC realization methods with high efficiency, which include a set of mathematical optimization models for limiting paths length, weight cut-off threshold, selecting constraint parameter and so on, and also include ant colony algorithm adapting to the characteristics of corresponding model. Finally, based on the edge detection of the record similarities, we try to establish a switching mechanism from FBS to RelDC, by representing the concepts of approximately duplicate, non-approximately duplicate, and suspect approximately duplicate using cloud model. According to the switching mechanism, the data cleaning framework, which is FBS-RelDC, will be constructed.

数据质量问题正在严重制约信息化建设进展，数据清洗是提高数据质量的重要技术手段。基于特征相似度(FBS)的方法和基于关系的数据清洗(RelDC)是当前主要的两种领域无关数据清洗方法。FBS具有效率优势，但存在相似度函数精度低、特征选择算法性能差、不宜处理复杂数据记录等问题；RelDC具有效果优势，但实现复杂度过高，且不能实现与FBS之间的适当切换。本项目通过定义不同属性类型的相似度函数并确定其参数，引入非线性分类器设计相似重复记录分类检测流程，建立特征选择数学模型并用蚁群算法求解模型，提出FBS新方法；通过建立RelDC的路径长度限制、权重修剪阈值、约束参数选择等优化数学模型，并设计适应模型特点的蚁群算法，提出高效的RelDC实现方法；通过用云模型描述相似重复、不相似重复、疑似相似重复等概念，实现记录相似度的边缘检测，建立FBS到RelDC的切换机制，进而构建FBS-RelDC数据清洗框架。

数据质量问题正在严重制约信息化建设进展，数据清洗是提高数据质量的重要技术手段。基于特征相似度(FBS)的方法和基于关系的数据清洗(RelDC)是当前主要的两种领域无关数据清洗方法。FBS具有效率优势，但存在相似度函数精度低、特征选择算法性能差、不宜处理复杂数据记录等问题；RelDC具有效果优势，但实现复杂度过高，且不能实现与FBS之间的适当切换。本项目提出了基于倒排索引消除冗余记录对、基于空间映射的数据块约减、基于Canopy聚类的数据分块方法，为提高实体分辨效率提供了方法基础；开展了实体分辨中的相似度计算方法研究，提出了基于多编辑距离融合的相似度计算方法和基于函数依赖的属性相似度调整方法，提高了实体分辨中的相似度计算准确性；提出了基于蚁群优化的特征选择方法和基于邻域粗糙集的快速特征子集选择方法，减小了参与实体分辨的属性规模，避免了干扰属性的引入，提升了实体分辨效率和效果。提出了基于关系类型的自适应实体分辨方法，解决了基于关系的实体分辨在计算路径权重时对训练数据要求过于严格，计算路径概率时不能区分关系类型的问题。提出了基于云模型的记录对划分方法和基于邻域粗糙集的实体分辨记录对划分方法，两种记录对划分方法实现了易分记录对和难分记录对的划分，实现了基于特征相似度的方法到基于关系的数据清洗的合理切换。提出了基于XML的数据质量检查规则表示与存储方法和基于ETL流程的数据质量控制规则模型，构建了数据质量框架，设计实现了数据质量控制系统，并在典型场景进行应用。