基于数据密集指数模型的大尺度遥感数据处理任务调度与执行优化

.Exploiting global or large area data sets as input for large scale remote sensing data processing have been receiving increasing attentions in global resource and climate change as well as regional to global research. When scaling to large area, remote sensing data processing becomes extremely challenging for the dependency relationships among a large collection of tasks that give rise to ordering constraint and complexity lies in parallel processing procedure, the demand of significant processing capabilities and also the difficulties inherent in intensive throughput of massive remote sensing data with high dimensionality instantaneously. As a result, traditional parallel remote sensing algorithms are not capable of large scale even global processing. To solve the problems above, we propose a dynamic task-tree based DAG scheduling and runtime optimization for large-scale remote sensing applications with data-intensive index model. Based on the constructing of data-intensive index model for remote sensing application, we propose a data-intensive aware optimized dynamic DAG scheduling for large-scale remote sensing task-tree, so as to decouple the dependency among tasks with parallel processing procedure. For enabling intensive loading and residing large remote sensing data, we propose a hierarchical distributed global remote sensing data model with data access pattern aware data layout optimization scheme. This hierarchical data model could offer hierarchical data layout controlling in distributed memory spaces for high degree of data locality. With these optimization schemes, we could offer high data throughput for massive remote sensing applications from runtime system level and as well as large-scale processing capacity. The study conduct in this project has a very important practical significance for promotion of global resources and environmental monitoring as well as relevant researches.

结合全球或大区域范围的遥感数据集进行大尺度遥感数据处理，已成为地球资源与全球环境监测的重要手段。在大区域甚至全球尺度下，数据密集计算问题凸显，结合面向计算密集集群的遥感数据并行处理仍极具挑战性：1）巨大的计算能力需求；2）海量多维遥感数据的瞬时密集吞吐；3）上千甚至几十万量级具有数据依赖关系的处理任务使并行处理逻辑极为复杂，传统并行算法在全球或大区域尺度下仍然难以适应。针对以上问题，本项目提出基于数据密集指数模型的大尺度遥感数据处理任务调度与执行优化研究。该研究基于数据密集指数模型的构建，研究数据密集程度感知的大规模任务树动态DAG调度策略，以及结合访问模式感知的分布遥感数据内存模型和布局优化的大数据缓存技术，通过将任务依赖关系与并行算法解藕并提高数据局部性，在运行系统支持海量遥感数据的高度密集吞吐，解决大尺度处理问题。本研究对于推动全球资源与环境监测及研究具有非常重要的现实意义。

结合全球或大区域范围的遥感数据集进行大尺度遥感数据处理，已成为地球资源与全球环境监测的重要手段。在大区域甚至全球尺度下，数据密集计算问题凸显，结合面向计算密集集群的遥感数据并行处理仍极具挑战性：1）巨大的计算能力需求；2）海量多维遥感数据的瞬时密集吞吐；3）上千甚至几十万量级具有数据依赖关系的处理任务使并行处理逻辑极为复杂，传统并行算法在全球或大区域尺度下仍然难以适应。针对以上问题，本项目提出基于数据密集指数模型的大尺度遥感数据处理任务调度与执行优化研究。该研究基于数据密集指数模型的构建，研究了数据密集程度感知的大规模任务树动态DAG调度策略，以及结合访问模式感知的分布遥感数据内存模型和布局优化的大数据缓存技术，通过将任务依赖关系与并行算法解藕并提高数据局部性，在运行系统支持海量遥感数据的高度密集吞吐，解决了大尺度处理问题。本研究对于推动全球资源与环境监测及研究具有非常重要的现实意义。本研究资助发表SCI期刊论文3篇，培养博士生两名，硕士生2名，项目投入经费25万，支出25万，预算与支出基本相符。