云存储中的协同纠删码技术

With the rapid growth of cloud storage center, the cumulative volume of data reaches EB and even ZB from PB. As a result, both network size and the number of storage nodes continue to grow explosively, while the data failure rate is still increasing. Cloud storage centers generally encode the cold data into erasure codes, so as to save the system overhead as much as possible meanwhile guarantee the reliability of data. However, the state-of-art erasure codes techniques still rely on conventional centralized model which results in unaffordable encoding/decoding cost, and thus cannot adapt to the data-intensive processing requirements for distributed cloud storage environments. . Aiming at the ways that the cold data is processed, this project is devoted to researching efficient collaborative erasure codes technologies, and focuses on the parallel encoding/decoding mechanism for complex network environment: this project proposes fundamental cooperative coding schema, which is able to ease the local network node performance bottleneck by allocate the intensive coding tasks to distributed nodes that participate in completing the data coding tasks collaboratively; Thereafter, this project proposes collaborative erasure codes technologies based on multiple data copies, which is able to coordinate several relevant nodes to archive the cold data copies in parallel; Furthermore, this project proposes the adaptive erasure coding technique, which can perform an adaptive selection of optimal nodes for cooperative coding in complex network environment; This project proposes a collaborative erasure coding technique based on the random data allocation, which is able to reduce the network traffic for data recovery; What's more, this project builds a prototype to evaluate the proposed technologies.

随着云存储中心的快速发展，存储数据量从PB到EB甚至到ZB，网络规模和存储节点数量呈爆炸式增长，数据失效率却越来越高。云存储中心往往将冷数据编码转换为纠删码分块，从而在保障数据可靠性的同时尽可能节省存储系统开销。然而，纠删码技术仍然依赖于传统的集中式处理模式，编/解码的系统开销大，难以适应云存储环境下的数据密集型处理需求。. 项目针对云存储系统的冷数据处理特点，研究高效的协同纠删码技术，主要从编码机制层面研究复杂网络环境下节点间的并行编/解码技术：研究基本协同编码模式，将密集的数据编码任务分配到多个节点协同完成，缓解局部网络性能瓶颈；研究面向多副本的协同纠删码技术，协调多个节点并行地将冷数据副本归档为纠删码分块；研究自适应协同纠删码生成技术，能在复杂网络拓扑下自适应地选择最优节点进行协同编码；研究基于随机分组的协同纠删码技术，降低数据恢复的网络开销；并搭建原型系统，综合评估所提技术。

项目从混合云存储平台体系结构及支撑系统的特性研究入手，对分布式网络存储系统中的固态节点失效特性以及整个云存储网络系统内节点的失效特性进行了研究，在归纳多个失效或随机差错影响参数的基础上，总结出在多种系统物理参数影响下的节点间的非均匀失效特性，并建立了可靠性参考模型。可用于解决大规模分布式网络存储系统的高节点失效率问题，有效降低系统的恢复开销，提高数据访问的吞吐量，提高系统性能。代表性研究成果如下：. (1)研究了适合分布式存储系统和固态存储系统的低功耗的多种纠删码技术。其中，DERS编码算法可用于缓解固态存储系统中LDPC纠错算法的错误翻转扩散问题，减少循环迭代次数，有效提高系统纠错能力和性能；权值多比特翻转LDPC译码方法可用于优化闪存系统的LDPC纠错编码性能，有效提高系统吞吐率和纠错能力；针对云存储网络环境下的非均匀失效特性，提出了UFP-LRC纠删码技术，能有效降低系统恢复开销，提高云存储系统性能；提出一种针对NDN 缓存的基于请求频率的实时数据替换方法，可用于优化NDN网络系统的缓存效率，提高缓存数据的命中率，提高系统性能。. (2)针对云存储数据的访问频率不均匀的特点，提出海量冷数据扫描抽取技术，通过抽取出已经冷却的数据，再纠删码编码操作，可用于在保证数据可靠性和少量增加计算与通信开销的前提下，大量缩减存储开销。. (3)研发了能对海量数据进行纠删编码处理的原型存储系统，并成功用于省公安厅电子政务大数据安全管理平台。