键值存储系统架构设计与性能优化研究

With the rapid growth of unstructured data, conventional block-based and file-based storage systems fail to satisfy the application requirements in both aspects of scalability and performance, so key-value store is widely deployed. Most widely used key-value storage systems mainly adopt the structure of LSM-Tree, but it introduces severe write amplification and read amplification problems, which imply that each insert or query operation of a key-value pair may incur multiple I/O operations, and they significantly reduce the read and write performance of key-value systems. Evenworse, when storage coding is employed to improve system reliability, parity updates may exacerbate the write amplification issue. To solve the above issues, we aim to develop a new architecture for key-value storage systems so as to improve both the performance and reliability. Specifically, we follow the idea of key-value separation, and propose a structure of fine-grained levels via grouping for LSM-Tree and a log-structured container management scheme to manage keys and values, respectively, so as to optimize the write performance. We also exploit the access patterns of data, and develop bidirectional LSM-Tree and heterogeneous bloom filters to optimize the read performance. Besides, we introduce storage coding and optimize the management of redundant data, so as to simultaneously improve both the performance and reliability. At last, we develop a system prototype, and run benchmarks to validate the system performance.

随着非结构化数据的快速增长，传统的块存储和文件存储在可扩展性与性能方面均无法满足应用需求，因此键值存储受到广泛应用。当前主流的键值存储系统主要采用LSM-Tree结构，但该类系统也面临严重的读写放大问题，也即每写入/查询一个键值数据可能引发多次I/O，严重降低键值系统的读写性能。此外，当引入存储编码以提升系统可靠性时，校验数据的更新将进一步加剧写放大问题。鉴于此，该项目计划建立键值存储系统新型架构，以优化系统的读写性能与可靠性。特别地，我们计划采取key和value分离的思想，建立基于层组结构的LSM-Tree和基于日志结构的容器管理技术对key和value分别进行管理，以优化系统写性能；同时结合数据访问特征，建立双向LSM-Tree和异构布隆过滤器优化系统读性能；此外，我们引入存储编码并优化冗余数据管理，以同时提升系统性能与可靠性。最后，通过开发原型系统，测试并验证系统的性能。

项目针对键值存储系统开展了深入研究，重点围绕基于LSM-tree的键值存储系统，对其读放大问题、写放大问题、读写均衡问题、可靠性管理问题开展了深入的研究。具体而言，针对读放大问题，提出了基于冷热感知的弹性布隆过滤器管理技术；针对写放大问题，提出了基于哈希的数据分组存储管理技术；针对读写均衡问题，提出了基于LSM-tree和哈希的键值存储统一索引技术以及基于键值分离的差异化键值数据存储技术；针对可靠性问题，首次提出了基于副本解耦的多副本差异化存储技术。以上技术有效提升了键值存储系统的读写性能，同时保证了系统的高可靠。..项目产出方面，共发表/录用论文19篇，包括CCF推荐的A 类会议论文7篇（ATC 4篇，ICDE 2篇，FAST 1篇），A类期刊论文5篇（TPDS 3篇，TOS，TCAD各一篇）, B类论文4篇。均已标注项目资助。申请专利3项。键值存储部分成果在PingCAP落地应用，获得优秀合作奖。同时也得到了华为云的支持，签署了新的键值存储合作研究项目。