面向分布式存储系统的数据快速修复纠删码关键技术研究

In distributed storage systems, the erasure code ensures the reliability of the data with high storage efficiency and reduces the risk of data loss. The process of rebuilding the failed data is called data repair. The traditional erasure codes take up a lot of computing resources, network bandwidth and disk I/O in the data repair process. The fast-data-repair erasure codes can significantly reduce the repair cost, but they have some limits: only applicable for some specific coding parameters, sacrificing the fault tolerance, or only applicable for specific finite fields. To address above problems, the project will study the coding extension technology of fast-data-repair erasure codes from the following aspects: First, we study a class of extension method for coding parameters, applicable to all kinds of erasure codes, which can make the erasure codes suitable for general distributed storage systems, and maintain the fast repair ability. Second, we study some fast-data-repair erasure codes over specific Galois field, and measure their failure tolerance in the extended composite Galois field. At last, based on two above extension methods, we utilize the evolutionary algorithm and artificial neural network method to find erasure codes with high failure tolerance and fast repair ability, from the code-extended code set of above two processes.The results of this project will help to reduce the data repair cost and enhance the data reliability in distributed storage systems, while promoting the application of artificial intelligence technologies in the field of erasure code construction at the same time.

在分布式存储系统中，纠删码以较高的存储效率保证了数据的可靠性，降低了数据丢失风险。对失效数据进行重建的过程叫做数据修复，传统纠删码在数据修复过程中占用大量计算资源、网络带宽和磁盘I/O。数据快速修复纠删码能够显著降低数据修复开销，但现有该类编码存在局限性：编码方法仅适用于特定参数，牺牲容错能力，编码方法基于特定有限域等。针对以上问题，本项目将从以下几个方面研究这类纠删码的编码扩展技术：首先，研究适用于一般纠删码的编码参数扩展方法，使得其适用于通用的分布式存储系统，并保持原有的数据快速修复能力；其次，针对特定有限域上具有最优数据修复性质的纠删码，研究其在扩展后的复合域上的容错能力；最后，在经过编码扩展后的纠删码集合中，运用演化算法和神经网络筛选出具有高容错、高修复能力的编码方法。本项目成果将有助于减小存储系统的数据重建效率，提高系统可靠性，并有助于推动人工智能技术在纠删码编码构造上的应用。

纠删码因为其良好的容错性质和存储效率被广泛地运行在分布式存储系统中，但它也存在着修复开销大的问题。近年来，针对减少单节点修复开销的新型纠删码被不断提出，但这些编码方法的构造或基于特定方法，或仅适用于特定参数。总结新型纠删码减少修复开销的规律，提出构建通用编码方法对提高数据可靠性和推广减少修复开销编码方法的适用性具有重要意义，同样也能产生重大经济效益。..本项目研究在二元域GF(2)上分析了生成矩阵的最优容错性质和修复性质，提出一种在判定生成矩阵最优容错的简单判据，同时也提出了一种判断减少修复开销的快速验证方法，该方法保证生成矩阵扩展至一般有限域也维持了其修复开销。分别基于有限域GF(2^w)和二元域GF(2)，我们开发了rlclib编码库和brmlib编码库，并结合已有的减少修复开销的编码方法，验证了我们关于生成矩阵容错性质和修复性质的分析。我们还提出一种结合多核CPU和FPGA的异构并行计算框架，加速筛选生成矩阵和计算其修复开销的过程。