实时数据流处理系统中负载均衡与容错机制的融合方法研究

The data are time-sensitive in many areas, which means the data are valuable only when they are in the "fresh" state. In the recent years, the research on Distributed Stream Processing Engine(DSPE), which is used for processing large-scale real-time data, has become an indispensable part in the development of data science and big data technology. However, the intrinsic characteristics of stream data, such as its continuity, dynamic, and unpredictability, bring great challenges in the design of a high-performance and high-availability DSPE. .This project focus on the current issues in DSPE, such as the lack of algorithm which produces a node-wise lightweight adaptive workload balancing, the high cost of resilient processing architecture and the high cost of fault tolerance, pursuing a win-win fusion strategy of load-balancing and fault-tolerance mechanism. In particular, our research in this project will include the following several aspects: (1)A fast and data position sensitive workload balance theory according to the incoming data and task instances; (2)A light-weight workload deployment and adjustment strategy in the fusion method of load-balancing and fault-tolerance mechanism under dynamic workload; (3) The data backup and recovery mechanism under a specific data partition scheme of workload balance. We believe the outcome of this project will help to build a theoretical basis for the future real time system design. It could also be applied to the existing systems to improve the system performance, in terms of the throughput and processing latency, against dynamic workloads. Overall, the project is of great value both theoretically and practically.

在众多领域，数据具有时效性，即数据的价值更多聚集于其"新鲜"状态下。近年来，对用于海量数据实时处理的分布式数据流系统研究已经成为数据科学与大数据技术发展不可或缺的部分。然而，数据流的持续、动态及不可预知等固有特性给数据流系统的高性能、高可用机制设计带来了极大挑战。在此背景下，本项目以如何使负载均衡与容错机制“互利共赢”地共存于系统为导向，针对当前数据流处理系统中轻量级自适应负载均衡算法匮乏、弹性处理架构代价大及容错成本高等问题，拟设计一套同时实现处理节点高性能和系统高可用的均衡容错融合方案。研究内容包括：（1）根据数据及节点负载情况快速、位置敏感的均衡负载分配理论；（2）动态负载下，均衡容错融合机制中轻量级负载部署、调整方案；（3）均衡负载限定数据划分方式下的数据备份/恢复机制。项目的研究成果既为相关实时系统的设计提供理论基础，又可直接应用到现有系统中，具有较高的理论研究和实际应用价值。

随着信息技术的飞速发展，海量数据的及时处理分析需要日趋突出。海量实时数据处理技术能够助力更大程度地发掘数据价值。本项目以研究能够以较小资源开销为代价，保证实时数据流处理系统持续正确地处在高性能处理状态的技术为目标，分别探索了实时数据流处理系统中的并行处理的高性能方案、弹性架构的低代价方法及处理节点的强容错策略。基于此目标，本项目通过（1）负载均衡的理论保证，（2）动态负载的均衡策略；（3）均衡敏感的容错机制三个方面的研究内容展开工作。本项目的研究成果可以被直接应用到基于并行架构的各类实时数据流处理系统上，能够较显著地提升现有系统性能，进而催生更多大数据应用移植到实时处理系统上，让更多行业可以借助实时大数据处理来发掘聚集于数据“新鲜”状态下的价值。