动量项学习算法的确定型收敛性及应用研究

In this project we are mainly concerned with learning algorithms with momentum terms in neural networks. We analyze their deterministic convergence and make further applications for momentum terms, that is, we add momentum terms to the algorithms for solving singular linear systems and problems of numerical optimization. The main contents which will be studied in this project include: learning algorithms with momentum terms for feedforward neural networks with hidden layers and Sigma-Pi-Sigma neural networks; splitting iterative methods with momentum terms for singular linear systems; Krylov subspace methods with momentum terms for singular linear systems; and iterative methods with momentum terms for problems of numerical optimization. The research results of this project will give the deterministic convergence law for stochastic learning algorithms and make new bases for deterministic convergence analysis of learning algorithms in neural networks. At the same time, we introduce the idea of momentum updating to the iterative algorithms for solving singular linear systems and problems of numerical optimization, accelerate their convergence. So, the research results of this project will also provide new thoughtway for solving singular linear systems and problems of numerical optimization. The work in this project has many interdisciplinary original properties. By the study in this project, it will make the research in different fields such as neural network, numerical algebra and numerical optimization further come close to each other.

本项目主要是关注神经网络中带动量项的学习算法, 研究动量项对确定型收敛性的影响，同时对动量项的应用作进一步的拓展，即把它应用于奇异线性方程组和数值优化问题的求解。研究内容主要有：带隐层前馈神经网络和Sigma-Pi-Sigma神经网络的带动量项学习算法；奇异线性方程组的带动量项的分裂迭代法；奇异线性方程组的带动量项的Krylov子空间方法；数值优化问题的带动量项迭代算法。研究结果预期将揭示随机型学习算法的确定型收敛规律，以此对神经网络学习算法的收敛性分析奠定新的基础；同时我们对求解奇异线性方程组和数值优化问题的迭代算法，引进动量项更新的思想，实现对算法的加速收敛，进而为奇异线性方程组和数值优化问题的求解提供新的思路。本项目的研究工作具有较强的交叉性和原创性，通过对本项目的研究，将促成神经网络、数值代数以及数值优化这些不同领域的研究进一步融合渗透。

通过项目组成员的努力工作，项目组完成了本项目的研究工作，并取得了较好的研究成果。本项目主要是关注神经网络中一类带参数的学习算法,参数主要有学习率和动量项参数等，研究参数对确定型收敛性的影响，同时对参数的应用作进一步的拓展，即把它应用于大规模线性系统问题和数值优化问题的求解。研究内容主要有：神经网络的含参数学习算法；大规模线性系统的含参数分裂迭代法；大规模线性系统的含参数Krylov子空间方法；数值优化问题的含参数迭代算法。我们研制广义定常迭代和带奇异预处理子的Krylov子空间方法的数值算法，并进行算法的收敛性分析。我们关注迭代法的参数加速思想，特别是对动量项参数加速，我们解决了有关最优参数问题从而使迭代收敛达到最快。受本项目的基金资助，我们的研究成果主要包括十一篇学术论文(其中已发表七篇、已正式录用即将发表二篇、尚处于审稿中二篇)，两次国际学术会议的分组报告、一次国际学术会议的大会报告和一次国内学术会议的大会报告。另外，基于本项目的前期研究工作积累，在本项目的执行期间内，受本项目的基金资助我们还培养了硕士研究生十二名，其中已毕业六人，在读六人。