云计算平台上多类型任务的自适应调度技术研究

Cloud computing environment has the typical characteristics of dynamic, heterogeneous and massive multi-type task concurrency. Complexity control of task scheduling and adaptability of scheduling methods are the key to solve the task scheduling problem in the cloud computing environment. This project aims to solve the multi-objective constrained problem of Cloud computing by drawing lessons from QoS multi-objective constrained scheduling theory and taking the advantage of combinatorial optimization and machine learning. It also will offer new scheduling systems, which have the ability to adapt to cloud computing environment and to search global optimization and to find the efficiency of the combined benefits, and to effectively solve the complexity control problem of multi-objective constrained task scheduling．In addition,focusing on the problem that the type of tasks and scheduling algorithms may not accurately mapped,the project proposes an adaptive scheduling framework and a structure mechanism model in the Hadoop platform. The model can be adaptive to different scheduling algorithms according to the type of tasks, improving the accuracy of cloud computing platforms, enhancing multi-task scheduling efficiency, and realizing self-adaptive scheduling. This project provides technical support for scheduling theory under the constraints of the cloud computing QoS and adaptive scheduling mechanism in the Hadoop platform. It has significant theoretical and academic value for cloud computing environment.

云计算环境有动态性、异构性和海量多类型任务并发性等典型特征，而其任务调度的复杂度控制和调度方法的自适应性问题，是解决云计算环境任务调度的关键问题。本项目针对云计算环境中任务调度多目标（如成本、效率、客户QoS和完成时间等因素）约束的问题，借鉴QoS多目标约束调度理论，融合组合优化和机器学习的各自优势，提出适应于云计算环境下的，全局寻优能力和寻找效率组合优势较高的调度体系，有效解决多目标约束的任务调度复杂性控制问题。其次，针对任务类型和调度算法不能准确映射的问题，项目中提出Hadoop平台下的自适应调度框架和结构机制模型。该模型可以根据任务类型自适应切换不同的调度算法，有效提高云计算平台的多类型任务调度效率和调度准确率，实现调度的自适应性。项目为云计算的QoS约束下的调度理论和Hadoop平台下的自适应调度机理提供技术支撑，对云计算环境下的任务调度具有一定的理论意义和学术价值。

摘要：在动态异构海量任务的云环境中，对资源任务合理调度算法的研究是当前国内外研究的重点。本项目借鉴QoS多目标约束调度的理论，并融合组合优化和机器学习的优势，对保证用户满意度的前提下的资源调度和任务分配的各类算法做了深入研究。我们研究了Hadoop平台下任务资源的调度机制，现有调度算法的参数配置方式，提出了一种自适应资源的监测模型，通过对资源的动态监测，根据当前的资源分配情况选择合适的分配方案，获得了一系列的成果；利用监测模型的成果，提出了利用决策树C4.5算法作业分类方法，通过对作业的“切分”，得到大规模作业的简单集；提出结合模糊商空间聚类和改进遗传算法的资源任务分配模型，根据资源本身性能聚类，在聚类中基于改进遗传算法进行分配，极大提高了用户时间和资源满意度；针对MapReduce框架下计算型调度器在处理存储型作业耗时长的问题，提出了基于模拟退火算法的queue级别调度策略，实现了更少的作业完成时间以及更高的资源利用率；通过该项目的研究提高了云计算效率、降低了云平台硬件资源投入总成本，对山西省科学技术和产业发展起到了积极的推动作用。