大数据分析引擎“系统配置”自动调优关键技术研究

Big data analysis engines such as Spark and Flink have been widely considered as important tools for efficient analysis of big data. However, in real scenarios, they often result in very inefficient or even failed operations due to improper system configuration. Our previous research found that there are four major challenges in the optimization of system configuration for big data analysis engines: a large number of configuration parameters with complex non-linear dependencies between the parameters; the processed data will affect the optimal system configuration; different characteristics of big data programs will affect the optimal system configuration; configuration optimization for different big data analysis engine require different modeling methods. To address these four challenges, we propose the key technology for automatic tuning of the "system configuration" of the big data engine. The main research contents include: large data program classification method based on feature learning; performance modeling techniques for high-dimensional configurations with small samples; accurate parameter importance and interactivity analysis with small samples; high-efficiency search algorithm of high-dimensional configuration parameters; unified configuration framework of big data analysis engine. The research results will greatly improve the performance of big data analysis engine and solve the problem of modeling for a complex system with high-dimensional configurations and fewer samples. It will open up new ideas and provide methodological guidance for performance analysis, modeling, and optimization for big data analysis engines.

大数据分析引擎如Spark和Flink等已经被广泛认为是大数据高效分析的重要工具。然而，现实场景中它们常常因为不合适的系统配置导致非常低效甚至运行失败。我们的前期研究发现大数据分析引擎系统配置的优化问题中存在四大挑战：配置参数数量多且参数之间存在复杂的非线性依赖；处理的数据会影响系统的最优配置；不同大数据程序的特征会影响系统的最优配置；不同的大数据分析引擎优化配置需要不同的建模方法。针对这四个挑战，提出大数据分析引擎“系统配置”自动调优关键技术研究。主要研究内容为：基于特征学习的大数据程序分类方法；针对高维配置小样本的性能建模技术；小样本条件下精确的参数重要性和可交互性分析；高维配置参数的高效搜索算法；统一的大数据分析引擎配置框架。课题研究成果将大幅提升大数据分析引擎性能，解决复杂系统建模面临的高维配置小样本的难题，为大数据分析引擎的性能分析、建模和优化开拓新思路和提供方法论指导

大数据分析引擎如Spark和Flink等已经被广泛认为是大数据高效分析的重要工具。然而，现实场景中它们常常因为不合适的系统配置导致非常低效甚至运行失败。我们的前期研究发现大数据分析引擎系统配置的优化问题中存在四大挑战：配置参数数量多且参数之间存在复杂的非线性依赖；处理的数据会影响系统的最优配置；不同大数据程序的特征会影响系统的最优配置；不同的大数据分析引擎优化配置需要不同的建模方法。针对这四个挑战，提出大数据分析引擎“系统配置”自动调优关键技术研究。主要研究内容为：基于特征学习的大数据程序分类方法；针对高维配置小样本的性能建模技术；小样本条件下精确的参数重要性和可交互性分析；高维配置参数的高效搜索算法；统一的大数据分析引擎配置框架。课题研究成果将大幅提升大数据分析引擎性能，解决复杂系统建模面临的高维配置小样本的难题，为大数据分析引擎的性能分析、建模和优化开拓新思路和提供方法论指导