云计算环境中的大数据在线聚集技术研究

With the development of information industry, the amount of data produced by enterprises and organizations is growing rapidly, big data analysis and mining have been gaining increasingly popularity in recent years. However, due to the data volumes and bandwidth limit of disk and memory, the big data analysis often takes long time to return the final results even in the cloud environment, which makes it unable to meet the requirements for real-time and interactive data processing. Many large scale aggregation queries are used to get a “big picture” of the data. Instead of computing the time-consuming precise result, the real-time approximately correct answer is of more commercial value. Thus, online aggregation on big data has been received considerable attention from the data management community in the past few years. In this project, we first propose the online aggregation system architecture, based on which we survey the state of art of online aggregation from four aspects: online data processing, data sampling, aggregation result estimate and data convergency diagnosis. Then, based on the characteristics of data organization and data processing in the cloud, this project aims at systematic exploring the basic theory and implementation methods for online aggregation of big data in the cloud. And we mainly focus on four key issues: online aggregation technology focused on ad-hoc complex queries on big data, self-adaptive online aggregation technology based on data distribution and workload characteristics, online aggregation for multiple queries. These studies will provide new ideas, theoretical methods and technical support for further research for online aggregation in the cloud.

随着信息产业的飞速发展，人们需要管理的数据量正以前所未有的速度增长，分析海量数据并挖掘其蕴含的价值成为最具增长性的业务之一。然而受到数据规模的影响、磁盘和内存带宽的限制，大数据分析即便是在云计算环境下，也无法达到实时处理及与用户交互的速度要求。对很多大数据分析应用来说，相对于耗费大量时间和计算资源所获取的精确结果，快速获得的具有准确性保证的估计结果更有意义。所以大数据的在线聚集逐渐成为当今国内外学者的研究热点之一。本项目首先依据在线聚集框架对在线查询处理技术、数据采样技术、聚集结果估计技术和收敛程度诊断技术的研究现状进行分析。在此基础上，本项目拟从云计算中大数据的存储和处理特质出发，具体解决大数据领域应用中面向即席复杂查询的在线聚集技术、感知数据分布和负载特征的自适应在线聚集技术、面向多查询场景的在线聚集技术等关键问题，为云计算中针对大数据的在线聚集进一步研究与应用奠定基础、提供新思路。

随着信息产业的飞速发展，各类组织和企业产生的数据量正以前所未有的速度增长，分析海量数据并挖掘其蕴含的价值成为最具增长性的业务之一。然而受到数据规模的影响、磁盘和内存带宽的限制，大数据的复杂分析即便是在云计算环境下，也无法达到实时处理及与用户交互的速度要求。对于很多大数据分析应用来说，与其耗费大量时间和计算资源获取完全精确的结果，快速获得的具有准确性保证的估计结果更有价值。在线聚集提供了一种根据样本快速获得估计结果及其“精确程度”的方法，具有重要的意义。本项目围绕大数据领域的在线聚集基础理论和关键技术，从以下五个方面展开了研究：1）基于马尔可夫链的多表复杂查询在线聚集技术；2）基于负载特征和数据分布特征分析的动态采样技术；3）面向多查询场景的在线聚集优化技术；4）在线聚集技术在可穿戴领域中的应用探索；5）面向数据分析应用的大数据系统性能评测分析。通过项目的研究，形成了覆盖数据管理平台、采样技术、在线查询处理、结果估计算法的较为完整的在线聚集多层次知识体系，为后续研究奠定了基础；将在线聚集技术在可穿戴计算领域进行了应用，为在多应用场景的扩展做出了贡献。