云计算环境下面向大数据的在线聚集并行优化机制研究

Online aggregation evolves the accurate query processing where data are completely scanned into sample-based approximate query processing, which is a essential data processing technology for big data, especially to those application where accuracy is not as important, such as tendency analysis, evaluation, prediction and other data processing scenarios. Current research work of online aggregation focuses on the deployment and implementation problems in cloud environment to ensure the functionality, but ignores the optimization on data organization and task execution concerning the cloud architecture. This project is dedicate to enhancing the performance of online aggregation in cloud environment, focuses on the sampling performance, data placement, concurrent multi-query and estimation failure problems which challenges the current performance of online aggregation in cloud environment, steps further into the mechanisms on data organization and management, concurrent multi-query optimization and query mode switch in the cloud PaaS layer and finally provides technical support for efficient big data approximate concurrent query. This project will implement the online aggregation prototype system in cloud environment that will be deployed on the Southeast University Cloud Platform and apply it to the big data analyze applications such as social network and e-commerce, which will flourish the big data research nationwide.

在线聚集是将面向数据完全扫描的精确查询计算转变成面向随机样本的近似查询计算，在当今的大数据时代，这是一个非常重要数据处理方法，尤其是对查询精度要求不是太高的应用，如趋势分析、评估、预测等数据分析场景中。目前，在线聚集已有的研究工作主要是在现有云计算架构下的部署与实现问题，确保其功能的可用性，没有从云计算架构下的数据组织、任务执行等方面对在线聚集进行性能优化。本项目以提高云环境下在线聚集查询的执行性能为目标，针对当前云环境下影响在线聚集查询执行性能的采样效率、数据放置、多查询的并发和估计失效等问题，深入研究云计算PaaS层面向在线聚集查询的数据组织管理、并发查询优化和查询模式切换的有关机理和机制，为大数据近似估计查询的高效并行计算提供技术支撑。本项目将实现一套云计算在线聚集原型系统，在东南大学云计算平台部署验证，并应用到社交网络、电子商务等大数据分析应用中，推动我国大数据处理的研究。

针对在Hadoop框架下，在线聚集查询作业被分解成若干子任务并指派到合适的节点独立完成各自的随机采样、统计量计算以及近似估计等过程时，没有考虑多查询任务之间的优化问题，首先研究了用户提交的多查询任务到Hadoop并行计算框架的翻译过程，在Hadoop并行计算框架作业级，制定作业合并规则，合并相似查询的作业；其次在Hadoop并行计算框架任务级，研究分析了建立相似查询作业的采样相关性与计算相关性理论模型，使之共享数据采样和统计量计算，减少冗余开销；最后，搭建云计算在线聚集系统的研究试验平台，进一步探究了基于内存计算的可行性。