大数据流式计算能耗模型及优化研究

Ignoring the issue of energy consumption in big data stream computing, the problem of high energy consumption generated from data processing needs to be solved. On the basis of performance constraint of big data stream computing platform and our previous work, the research contents of this project focuses on the following three aspects: energy consumption mechanisms, energy models and energy optimization methods for big data stream computing. Firstly, based on the research of energy consumption model of memory, CPU, network bandwidth, disk, and other components, by monitoring resource state while task running we study the energy consumption mechanism of different components and their interactions in big data stream computing environment. Secondly, based on the research of energy consumption mechanism, the model of energy forecast, energy monitoring, and energy settlement can be established based on sampling for big data stream computing before performing a topology. As a result, we can forecast the energy consumption of big data stream computing topology before execution, monitor the dynamic energy consumption while topology is running, and estimate the energy consumption value after the topology is completed. Finally, the optimization of energy consumption aimed at the topology execution of the whole big data stream computing, that is from optimizing the energy consumption of different components for big data stream processing as well as the allocation of resources, to the overall energy efficiency of big data stream processing. This research is expected to improve the overall energy efficiency of big data stream computing topology and cluster, supporting key technologies of power and energy management for big data stream computing.

由于大数据流式计算处理数据时缺乏对能耗问题的考虑，导致其数据处理过程产生的高能耗问题亟需解决。结合大数据流式计算平台的性能约束与前期工作，课题对流式处理的能耗机理、能耗模型及能耗优化三个方面进行研究。首先，在建立内存、CPU、网络带宽与磁盘等元件能耗模型的基础上，结合任务资源的分配与监控，研究大数据流式计算环境下不同元件自身及彼此之间的能耗机理；其次，在能耗机理研究的基础上建立能耗预测、能耗监控及能耗结算三种模型，实现对大数据流式计算拓扑任务执行开始前能耗的采样预测、执行过程中能耗的监控以及执行后能耗的结算功能；最后，能耗优化旨在贯穿整个大数据流式计算的执行，即从优化流式处理不同元件产生的能耗以及资源的分配，到优化流式处理的整体能效。研究结果有望整体上提高大数据流式处理拓扑及集群的能耗效率，形成对大数据流式处理能耗管理关键技术的支撑。

由于大数据流式计算处理数据时缺乏对能耗问题的考虑，导致其数据处理过程产生的高能耗问题亟需解决。结合大数据流式计算平台的性能约束与前期工作，课题对流式处理的能耗机理、能耗模型及能耗优化三个方面进行研究。首先，在建立内存、CPU、网络带宽与磁盘等元件能耗模型的基础上，结合任务资源的分配与监控，研究大数据流式计算环境下不同元件自身及彼此之间的能耗机理；其次，在能耗机理研究的基础上建立能耗预测、耗监控及能耗结算三种模型，实现对大数据流式计算拓扑任务执行开始前能耗的采样预测、执行过程中能耗的监控以及执行后能耗的结算功能；最后，能耗优化旨在贯穿整个大数据流式计算的执行，即从优化流式处理不同元件产生的能耗以及资源的分配，到优化流式处理的整体能效。研究结果整体上提高大数据流式处理拓扑及集群的能耗效率，形成对大数据流式处理能耗管理关键技术的支撑。