分布式流处理系统实时容错关键技术研究

In recent years, the requirement of realtime stream processing increases dramatically, which leads to the rapid development of distributed stream processing technologies. Distributed stream processing system (DSPS) becomes a new type of distributed system and draw wide attentions. The system scale DSPS increase rapidly. The complexity of stream processing jobs grows with the scale as well. These new developments cause the rasing attentions about the reliability issues of distributed stream processing systems. .Currently, most related works focus on fault-tolerant mechanisms and protocols based on tradition methods in distributed systems. They are lack of the ability to analysis the failure effects to realtime performance, lack of optimized fault-tolerant strateries, and not able to fit the dynamic characters of data streams. .This poposal presents a novel research on the key technologies of realtime fault tolerance of distributed stream processing systems. The proposed research focuses on the realtime performance of fault tolerence. We plan to conduct researches on theoretical models, optimization methods and supporting mechanisms. The proposed reasearch includes: 1) Realtime-performance-oriented node-level fault-tolerant cost model and system-level reliability model, 2) Fault-tolerant strategy optimization methods and dynamic adjustment problems, 3) Supporting mechanisms and protocols of realtime fault tolerace. Comparing with the existing works, the contributions of the proposed research will include: 1) introducing the fault-tolerant affect to realtime preformance into system reliability model, 2) the optimization of fault-tolerant strategies based on topology-aware node-level fault-tolerant cost, 3) fault-tolerant dynamic adjustment supporting auto-scale and burst streams. The proposed research has important theoretical values and extensive application requirements.

近年来，各行业应用中对大规模流数据进行实时处理与分析的需求激增，分布式流处理技术发展迅速。随着系统规模的扩展和处理流程的复杂化，系统鲁棒性问题日益得到关注，分布式流处理系统故障容错问题逐渐成为研究热点。现有工作缺乏面向处理实时性的容错理论模型，缺乏相应的容错策略优化方法，容错方法动态性不足难以适应流数据处理特点。本研究重点关注分布式流处理系统的实时性容错问题，从理论模型、优化策略及支撑机制等层面开展研究，主要研究内容包括：面向处理实时性的节点故障代价模型及系统可靠性模型、容错策略优化及动态调整问题、实时性容错支撑机制及协议等。与现有的工作相比，本研究主要特色包括：将故障对实时性的影响引入到可靠性理论模型中、基于拓扑感知思想根据节点故障代价优化实时容错策略、支持自动并行化和突发性流等场景的容错策略动态调整等。本研究具有重要的理论价值和广泛的应用需求。

随着各行业应用中对大规模流数据进行实时处理与分析的需求激增，分布式流处理技术发展迅速。这类分布式系统规模迅速扩展，数据处理流程日益复杂化，系统鲁棒性问题日益得到关注，分布式流处理系统故障容错问题逐渐成为研究热点。目前亟需建立面向处理实时性的容错理论模型，进而快速发展相应的容错策略优化方法。本项目从理论模型、优化策略及支撑机制等方面，全面开展流数据处理系统的实时容错关键技术研究。本研究首次建立了面向处理实时性的分布式流数据处理系统节点故障代价模型及系统可靠性模型；在该模型基础上，进一步建立了多种实时性容错策略，包括面向容错代价的容错策略、面向处理延迟的容错策略以及面向突发性流负载的容错策略等；最后，开展了实时性容错支撑机制及协议研究，基于开源分布式数据处理框架实现了实时容错流数据处理原型系统SPATE，对本项目的理论研究结果进行验证。与现有的工作相比，本研究主要特色包括:将故障对实时性的影响引入到可靠性理论模型中、基于拓扑感知思想实现了考虑节点故障代价的优化容错策略、支持自动并行化和突发性流等场景的容错策略动态调整等。本研究基于通用分布式流数据处理系统模型，相关成果可适用于各类流行的分布式流数据处理平台，如Storm、Flink等。本研究具有重要的理论价值和广泛的应用需求。