大规模机器学习的动量加速方法的理论研究与应用

This project focuses on the design and theoretical analysis of momentum accelerated algorithms for large-scale machine learning. We will propose a series of momentum accelerated algorithms for various structure optimization problems, which include the study as follows: 1) We will propose a momentum acceleration framework based on the removal of residuals and overcome the problem that the convergence rate is slow due to variance perturbation in stochastic optimization. In theory, the proposed algorithm achieves the optimal convergence rate. 2) The proposed momentum acceleration framework can be extended to various structural optimization problems (including constrained optimization problems, convex-concave saddle point problems, and so on). This framework can improve the convergence rate of stochastic ADMM algorithms and stochastic primal-dual methods. In particular, in the stochastic primal-dual hybrid gradient algorithm, we will propose the fusion of momentum and variable decoupling, provide the iterative rules of the dual and primal variables, and analyze the convergence rate of the algorithm. (3) For several types of non-convex optimization problems with special structure, several effective and simple methods with momentum acceleration and theoretical analysis are proposed.. A series of acceleration algorithms proposed in this project are simple and practical, and have strong scalability. The research will not only make some contributions to the study of constructing numerical algorithms for optimization problems, but also expand the application domain of the optimization problems into practical problems. The research of this project also provides technical support and theoretical basis for the further research of large-scale optimization.

本项目研究大规模机器学习的动量加速算法的设计与理论分析。为了求解不同的结构优化问题，提出一系列动量加速的随机优化算法。重点研究以下内容：1）对于无约束优化问题，构造一种余项移除的动量加速框架，可克服由随机方差引起的收敛变慢的问题，提出的算法在理论上达到最优的收敛率。2）我们将提出的动量加速框架推广到不同的结构优化问题中(如约束优化、凸凹鞍点优化等)，可提升随机ADMM算法以及随机原始对偶方法的收敛率。特别在原始对偶混合梯度算法中，将动量与变量解耦的思想融合，给出原始对偶变量的迭代规则，并分析该算法收敛性。3）针对几类特定结构的非凸优化，将提出几种简单有效的动量加速方法及其理论分析。. 本项目提出的一系列加速算法简单实用、可扩展性强，不仅对优化问题数值算法的构造有重要意义；还将扩大优化问题在现实问题中的应用范畴，为大规模优化问题的进一步研究提供技术支持和理论基础。

本项目研究大规模机器学习的动量加速算法的设计与理论分析。针对不同的结构优化问题，将提出一系列动量加速的随机优化算法。重点研究以下方面的内容：1）对于无约束优化问题，构造一种基于补偿的动量加速框架，克服随机优化中由于方差扰动而导致收敛率变慢，提出的算法在理论上达到最优的收敛率。2）我们将提出的动量加速框架推广到不同的结构优化问题中(如约束优化问题、凸凹鞍点问题等)，可提升随机ADMM算法以及随机原始对偶方法的收敛率。特别在原始对偶混合梯度算法中，将动量与变量解耦的思想融合，给出原始对偶变量的迭代规则，并分析该算法收敛性。3） 针对稀疏优化问题，如何设计快速有效的算法，并给出收敛率分析结果。4）设计具有理论保证的隐私保护的优化算法 5）设计基于可学习展开网络的优化算法.本项目提出的一系列加速算法简单实用、可扩展性强，不仅对优化问题数值算法的构造有重要意义；还将扩大优化问题在现实问题中的应用范畴，为大规模优化问题的进一步研究提供了技术支持和理论基础。