云环境下面向大数据并行计算的工作流执行优化研究

Nowadays, the common cloud platform focus on the big data analysis always consists of three components, that is the distributed file system, the execution engine for data parallel processing and the programmatic interface. Note that, the programmatic interface is responsible for mapping the input complex jobs into the workflow, which is processed by the execution engine. However, the state-of-art cloud platforms do not support the workflow scheduling mechanism so that the job priorities cannot be guaranteed. And the time-constrain does not considered in the task assignment, affecting the job performance due to the superabundant data locality; Moreover, the existing cloud platforms do not manage the intermediate results among jobs efficiently, increasing the I/O cost that is generated by accessing the intermediate results. To overcome the above problems, this project aimed at the optimization for workflow processing in the cloud and extends the existing cloud platform to support the workflow scheduling mechanism and guarantee the job priorities, and then we add the intermediate result management component to realize the memory-based intermediate result management, reducing the I/O cost between jobs. Afterwards, we study the appropriate parallel task assignment algorithm coupling with the workflow scheduling and the intermediate result management to improve the workflow performance. Finally, we implement our workflow scheduling system for the existing cloud platform to demonstrate the efficiency and effectiveness.

目前，针对大数据并行处理的云计算系统分为三层，分布式文件系统、数据并行作业执行引擎和编程接口，编程接口负责将复杂应用解析成工作流交由数据并行作业执行引擎处理。然而，当前的云计算系统不支持工作流的调度机制，不能保证用户作业执行的优先级；子任务指派过程也没有考虑工作流作业执行的时间约束，过分追求数据本地性影响了作业的执行性能；没有对工作流作业的中间计算结果进行有效管理，增加了中间结果访问的I/O开销。针对以上问题，本项目以优化云环境下面向大数据并行计算的工作流执行为目标，改进现有的系统架构，引进工作流作业调度，保证多数据输出的用户作业优先级；引入中间数据管理，研究基于内存的高效中间计算结果存储管理机制，减少后继作业读取中间结果的I/O开销；研究与工作流作业调度和中间结果数据存储相匹配的并行子任务指派方法，保证工作流作业的顺利执行。在此基础上，实现云计算工作流调度系统，验证本项目提出的相关理论。

主要针对云计算系统三层架构分布式文件系统、数据并行作业执行引擎和编程接口对大数据并行处理的性能影响问题，进行了深入研究，给出了性能提升改进的相关方法和模型。针对SQL查询到MapReduce作业流翻译器自动生成的MapReduce作业流的执行效率很低问题，给出一种基于执行代价的类SQL查询到MapReduce作业流的翻译器。该翻译器采用MapReduce作业的代价估算模型来选择由自顶向下和自底向上两种作业合并策略生成的MapReduce作业流中执行代价较小的，大大提高了MapReduce作业流的执行效率。基于分布式系统HDFS，设计了基于内存的HDFS系统架构，在对中间结果数据读写时，首先使用内存存储资源，提高了数据中间结果数据的读写速率。在底层HDFS与Spark核心间设计了存储中间层，实现了Spark SQL查询作业间的共享数据，从而提高了Spark SQL作业的执行效率。针对Spark现有数据分区器无法有效解决Shuffle阶段出现的数据倾斜所引发的Reducer负载不均衡问题，提出了一种新的数据分区器（DS-Partitioner），实现了Shuffle阶段中间数据的均衡分配。