高铁供电准实时大数据集群快速响应机制及其列压缩方法研究

High-speed rail power dispatching data center is characterized by the new features of elaborate data cluster, sub-second fast processing of quasi real-time big data is the largest volume, showing the geometric growth. Due to its growth rate far exceeds the server storage capacity, making traditional relational data storage facing the great challenge. Slow response to massive data easily leads to screen crash, resulting in late train delay recovery and other issues. Therefore, it is urgent to study the quasi-real-time response mechanism and the new method of large-rate compression. The project aims at basic scientific research on the big data of railway power supply. It includes: (1)Designing the inverted second-level index of non-main-line key data of railroad power supply stored by columns and rapidly locking the target data by combining the main line keys of map and specifying the adaptability to the delay of hundred milliseconds; (2)Task resource perception model to improve the real-time calculation of rail-based real-time computing cluster scheduling mechanism to obtain more balanced query response characteristics and application conditions; (3)The establishment of rail-powered data compression model, the column compression applied to the connection query Wash stage, research data aggregation query results. The successful implementation of this project aims at revealing the necessary conditions and laws for rapidly processing dispatched big data of railway power supply and accelerating the dispatching information processing speed, which has important theoretical significance and application value for reducing the late-point rate and guaranteeing the transport safety.

高铁供电数据中心的新特征是大数据集群处理，体量极大的全景准实时数据属于次秒级处理的类别，其几何级增长速度远超服务器存储容量增长，使传统关系数据存储受到极大挑战，海量数据响应慢易导致卡屏，造成列车晚点恢复不及时等问题，迫切需要研究准实时数据响应机制和大比率压缩方法。本项目针对铁道供电大数据展开基础科学研究。包括:(1)对按列存储的铁道供电非主行键数据设计倒排二级索引，组合映射主行键快速锁定目标数据，明确对百ms级延时的适应性；(2)研究任务资源的集群感知模型，改进铁道供电准实时计算集群的底层调度处理机制，获得更均衡的查询响应特性和应用条件；(3)建立铁道供电数据集群列压缩模型，应用到加速连接查询的混洗阶段，研究聚合端数据压缩查询效果。本项目成功实施旨在揭示快速处理铁道供电调度大数据所需的条件规律，加快调度处理速度，减少故障处理时间，对降低晚点率和保障运输安全具有重要理论意义和实用价值。

高铁供电数据中心的新特征是大数据集群处理，体量极大的全景准实时数据属于次秒级处理的类别，其几何级增长速度远超服务器存储容量增长，使传统关系数据存储受到极大挑战，海量数据响应慢易导致卡屏，造成列车晚点恢复不及时等问题，迫切需要研究准实时数据响应机制和大比率压缩方法。本项目针对铁道供电大数据展开基础科学研究。包括:(1)对按列存储的铁道供电非主行键数据设计倒排二级索引，组合映射主行键快速锁定目标数据，明确对百ms级延时的适应性；(2)研究集群资源的任务调度模型，改进铁道供电准实时计算集群的底层调度处理机制，获得更均衡的查询响应特性和应用条件；(3)建立铁道供电数据集群列压缩模型，应用到基于列式数据库内存的扁平化设计中，研究了一种内存数据的折叠压缩方法。本项目成功实施旨在揭示快速处理铁道供电调度大数据所需的条件规律，加快调度处理速度，减少故障处理时间，对降低晚点率和保障运输安全具有重要理论意义和实用价值。