面向大规模机器学习的高效优化算法研究

There are some disadvantages in the current popular optimization algorithms for large-scale machine learning—it is common to set the learning rate of optimization algorithms in a try-and-error way; the convergence rate of the algorithms is still slow. To tackle these problems, we deeply study and propose some novel and efficient stochastic gradient descent algorithms for large-scale machine learning, i.e., (1) We propose a new stochastic gradient descent algorithm with adaptive learning rate. Based on the large-scale data, we utilize the law of large numbers and the central limit theorem to compute the Lipschitz constant, and then we can obtain the appropriate learning rate. (2).We propose an adaptively-weighted-average-history-gradients stochastic gradient descent algorithm. This new algorithm will take the full advantages of history gradients to make the variance of stochastic gradients estimation smaller, and then the convergence rate will speed with no extra-large memory. (3) We propose a new adaptively-weighted-average stochastic gradient descent algorithm. We reduce the variance of stochastic gradients based on the history mean and variance of samples in mini-batch, and then faster convergence rate of the algorithm will also become faster.

现有大规模机器学习中的优化算法存在以下不足：需要通过反复try-and-error方式设置初始学习率；优化算法的收敛速率仍比较慢。针对上述问题，本项目深入研究并提出适合于大规模机器学习任务的高效随机梯度下降优化算法：(1)提出一种新的学习率自适应的随机梯度下降法，由大规模机器学习的自身特点为基础根据大数定律和中心极限定理，经由计算Lipschitz常数自动得到合适的优化学习率。(2)提出一种新的自适应加权平均历史梯度的随机梯度下降法，此新优化算法将充分利用历史梯度信息使随机梯度估计方差减小，从而在不增加内存需求的前提下加快了收敛速度快。(3)提出一种新的自适应的历史加权Batch Normalization法，基于历史随机mini-batch中的样本均值和方差信息直接减小随机梯度估计方差，最终达到加快训练优化收敛速度的目的。

针对目前大规模机器学习优化算法中两个主要问题：需要通过反复try-and-error方式设置初始学习率和优化算法的收敛速率仍比较慢；本项目从三个方向对的优化算法进行探索：1）研究一种新的学习率自适应的随机梯度下降法，以减少初始学习率的影响； 2）研究自适应加权平均历史梯度的随机梯度下降法， 加快训练了训练速率的影响； 3）研究从优化的角度改进Batch Normalization 从而加快训练优化收敛速度。在项目的支持下，培养硕士研究生4名，发表高水平论文3篇，已投高水平论文2篇。