大数据流式在线应用的拓扑感知与弹性调度理论及方法研究

In big data era, more and more scenarios will be processed by big data stream computing platform, such as stock market analysis, click streams analysis, traffic stream analysis, and emergency response, to name but a few. All these scenarios have distinctive features, such as high throughput, low latency, continuously online, elastic and reliable. Big data stream also shows distinctive characteristics, such as real time, burstiness, volatility, irregularity and infinity. Usually, when compared with batch data, big data stream is difficult to be processed in real time by big data batch computing platform. The practical experiences, researches and discussions about big data stream computing are not comparatively enough. How to build a scalable, distributed, stable big data stream computing system is great challenge and scientific problem. In traditional study of stream computing, most of works are based on database, the data rate is in small-scale, and data stream often come from a single source, all those features are significantly different with that in big data streaming scenarios. In this project, we aim to analyze characteristics of data stream in big data stream scenarios, and carry out the research of big data stream computing platform from the perspective of architecture, which includes the following aspects. (1) Proposing a fine-gained topology-aware mechanism for online applications of users, which can keep an application transfer into a task graph with a reasonable structure. (2) Proposing an elastic, adaptive online scheduling strategy, which can meet the resource scheduling needs in a continuously online environment. (3) Proposing a lightweight, fast fault tolerance method, which can meet the system timeliness requirement for fault tolerance in a big data stream computing environment. In short, this project focuses on the topology of an application, online resource scheduling, system fault tolerance, a solution with features of reasonable architecture will be given, all those can meet the needs of high throughput, low latency, continuously online, elastic and reliable in big data stream computing environments.

大数据的时效性日益凸显，流式应用场景越来越多，呈现出了高带宽、低延迟、长期在线等鲜明需求，这和相对成熟的大数据批量计算场景很不一样。然而，大数据流式计算的实践经验和理论研究相对较少。传统流式计算往往在数据库的基础上开展，处理小规模、单一数据源的应用场景，不能满足大数据时代流式计算场景的需求。鉴于此，本项目从系统体系结构的角度，结合具体的大数据流式计算架构，开展大数据流式计算场景中在线应用的拓扑感知与弹性调度理论及方法研究：（1）提出一种细粒度用户应用拓扑感知机制，实现用户应用转化为结构合理的有向任务图；（2）提出一种弹性、自适应的在线调度策略，满足长期在线场景下资源调度需要；（3）提出一种轻量级、快速的容错方法，满足系统对容错时效性的要求。通过对大数据流式计算中应用拓扑结构、在线资源调度、系统容错等内容的研究，完善大数据流式计算架构，满足大数据环境下流式计算中高带宽、低延迟的应用需求。

本项目从系统体系结构的角度，结合大数据流式计算架构，开展了大数据流式计算场景中在线应用的拓扑感知与弹性调度理论及方法研究。取得的代表性成果为：.（1）针对状态、运行时感知的大数据流计算面临的挑战：用户应用的状态轻量级精准感知同大数据流计算系统长期在线运行之间的矛盾，提出了一种状态、运行时感知资源分配策略。在细粒度层面上实现对用户应用拓扑结构中各个节点的状态特征进行了语义分析，实现了在长期在线运行环境中进行状态的轻量级精准感知。通过首次适应的原则实现对用户应用的快速部署，在在线运行过程中，实现对当前用户应用部署状态的感知，并通过对最大延迟节点的运行时感知，实现对用户应用的精准优化和在线系统性能的稳定，构建起多应用流计算环境中状态、运行时感知资源分配策略，形成了大数据流式计算系统Sra-Stream。.（2）针对弹性大数据流式计算面临的挑战：用户应用的拓扑结构的静态配置同数据流的持续、动态变化之间的矛盾，提出了一种弹性、在线资源分配策略。针对用户缺乏应用拓扑结构优化的专业知识，量化了应用响应时间、多用户公平性以及多维度数据流在线特征等指标，分析了拓扑节点的语义，实现了对用户所设计的静态拓扑的再次动态优化和在线调整。针对多用户、多应用的场景，构建了多用户、多应用的数据流调度模型，实现了公平的多用户、多应用在线资源的弹性分配，形成了大数据流式计算系统E-Stream。.（3）针对可靠流式大数据计算面临的挑战：用户对系统的实时性要求强，导致日志、检查点等传统容错策略不能满足其对时间的要求。流式计算环境的动态持续变化，也使得时间、空间复杂度较高的容错机制无法使用到该场景。提出了一种在保证用户响应时间情况下的轻量级可靠性感知的流式计算应用分配方法。通过响应时间和可靠性两个不同的维度，对应用的拓扑结构进行优化和调整，实现了在满足特定响应时间要求的情况下，达到流式计算环境可靠性的最大化，形成了大数据流式计算架构FTDG。