集群环境下基于内存的高性能数据管理与分析

With the increasing intension of market competition and the continuously development of enterprise informatization, it is hard for the conventional business intelligence systems to meet the requirements about the timely decision. The real-time business intelligence is then becoming a goal which more and more enterprises are pursuing. The great advance on computer hardware and architecture offers technical background for the real-time business intelligence. The project mainly will focus on the high performance data management and analytics based on in-memory cluster computing, which is expected to set a solid foundation for the efficient processing of the decision-making big data, taking full advantage of the progress on hardware, and taking the real life application into consideration. The major research topics are as followings. 1) High performance data storage with non-uniform access memory, including column-oriented dense packing storage, memory-sensitive data placement, and compression-aware data processing. 2) Fully using the heterogeneous computation resources, including DAG workflow parallel processing based on function-at-a-core strategy, interactive parallel processing based on processor-affined scheduling, session scheduling strategy, and load balance based on computation priority. 3) High availability of the building systems, such as fault-tolerant data set based on lineage, task recovery based on standby process. 4) The optimization on primary operators and execution plan for the specific applications, to achieve the ad hoc human real-time interactive analysis. The planned research conforms to the current applications and the development of the related technologies. It is of broad interests to the participants from academic and industries. The applicants have profound technical accumulation on the related areas, and have explored preliminarily on the proposed research plan, which ensure this project to be accomplished successfully.

随着市场竞争的加剧和企业信息化程度的提高，传统商务智能系统难以满足当前决策时效性的要求，实时商务智能已成为许多企业追求的目标，硬件和体系结构的发展为之提供了技术条件。本项目旨在研究集群环境下基于内存的高性能数据管理与分析技术，探索符合应用需求、充分发挥硬件效能的决策类大数据处理技术，为实现实时商务智能奠定基础。重点研究：1）非一致访问内存环境下的数据高效存储，包括列式密集存储、内存感知布局和压缩感知处理；2）大规模异构计算资源的充分利用，包括函数至核有向无环工作流式和处理器结合迭代式并行处理，以及会话调度策略和计算优先负载平衡；3）内存系统可靠性保障，包括基于世系的内存数据集容错、热备进程的任务快速恢复等；4）针对特定应用的基本算子和执行计划优化。本项目的研究符合现实应用需求和技术发展趋势，具有广阔的应用前景和学术价值。申请人在数据管理方面积累充分，研究方案可行，能保证本项目顺利完成。

随着市场竞争的加剧和企业信息化程度的提高，传统商务智能系统难以满足当前决策时效性的要求，实时商务智能已成为许多企业追求的目标，硬件和体系结构的发展为之提供了技术条件。本项目旨在研究集群环境下基于内存的高性能数据管理与分析技术，探索符合应用需求、充分发挥硬件效能的决策类大数据处理技术，为实现实时商务智能奠定基础。重点研究内容包括：1）非一致访问内存环境下的数据高效存储，包括列式密集存储、内存感知布局和压缩感知处理；2）大规模异构计算资源的充分利用，包括函数至核有向无环工作流式和处理器结合迭代式并行处理，以及会话调度策略和计算优先负载平衡；3）内存系统可靠性保障，包括基于世系的内存数据集容错、热备进程的任务快速恢复等；4）针对特定应用的基本算子和执行计划优化。本项目的研究符合现实应用需求和技术发展趋势，具有广阔的应用前景和学术价值。..项目针对研究目标，分别取得了如下创新性结果：1）针对集群内存访问的多级性，提出了融合缓存、内存、SSD存储、外存、远程数据访问的多级数据存取模型，实现了数据的高效存取。2）利用集群内大规模异构并行计算资源，提出了细粒度的弹性并行查询处理框架，实现了复杂分析任务子任务自动分配以及高效并行执行与高吞吐量调度。3）针对内存数据易失、复杂任务易溃的特点，提出了本地日志存储、远程基于Paxos的日志同步等容错策略，提升了系统的可靠性、可用性。4）面向商务智能应用的需求，提出了一系列的分析型查询处理算法与优化技术，以及特定的流数据分析和图数据分析处理算法，实现了即席的交互式分析。..项目资助下，项目组成员发表期刊论文22篇，学术会议论文21篇，出版学术专著1部，申请专利8项，其中已授权2项，申请软件著作权5项，获得科研奖励2项。项目通过开源项目Gingko实现了成果的集成和应用示范，验证了成果的先进性和实用性。