深度学习中的低秩矩阵优化的模型及算法研究

Artificial Intelligence has made a revolutionary breakthrough with the help of deep learning. The Model of deep learning is very complex, which demands high storage space and computing resources. Model compression skills can remove redundant nodes in deep learning so that the consumption of computing space and resources can be effectively reduced. There are a lot of fresh low-rank matrix optimization problems in the neural network compression. However, optimization problems in deep learning are usually large-scale. The existing algorithms of low-rank completion usually fail to output an optimal or approximate solution within a reasonable time. How to effectively train the compressed neural network and exactly solve the low rank matrix optimization problems is a leading but challenging topic, which deserves further explorations. This project mainly focuses on the key technical problems, such as weighting parameter redundancy, high amount of calculation, and large scale. The main purpose is to achieve the effective training of neural network and try to solve the low-rank matrix optimization problems, so that promote the intersection of optimization and other disciplines. This project intends to establish models and algorithms of low-rank matrix optimization in deep learning. As for the large-scale problem, we can reduce dimension by the stochastic method, and design randomized algorithms for large-scale low rank matrix optimization in deep learning.

深度学习使人工智能产生革命性突破。但是深度学习的模型比较复杂，需要高额的存储空间和计算资源。模型压缩技术可以删除冗余的节点，从而减小深度学习对于计算空间和资源的消耗。模型压缩蕴藏了许多新型低秩矩阵优化问题。深度学习中的优化问题具有大规模的特点，现有的低秩矩阵恢复的模型和算法往往无法在合理时间内给出问题的最优解或近似解。如何有效训练压缩后的神经网络又求解低秩矩阵优化问题是一项有挑战性的前沿课题，需要进一步的探究。本项目重点关注神经网络中权重参数冗余、高计算量、问题规模大等关键技术难题，实现有效训练神经网络且求解新的低秩矩阵优化问题的目的，促进最优化与其他学科的交叉融合。本项目拟建立深度学习中低秩矩阵优化的模型，设计快速算法；针对大规模低秩矩阵优化，利用随机方法降维，为大规模低秩矩阵优化问题设计随机优化算法。

深度学习使人工智能产生革命性突破。但是深度学习的模型比较复杂，需要高额的存储空间和计算资源。模型压缩技术可以删除冗余的节点，从而减小深度学习对于计算空间和资源的消耗。模型压缩蕴藏了许多新型低秩矩阵优化问题。深度学习中的优化问题具有大规模的特点。本项目旨在开展低秩矩阵优化问题的模型和算法研究，获得如下三个方面的结果：(1) 为带线性约束、目标函数不可分离的凸优化问题设计了基于增广拉格朗日方法的半光滑牛顿算法，并给出了收敛性分析；(2) 为带线性约束、目标函数不可分离的凸优化问题设计了交替方向乘子法，并给出了收敛性分析；（3）基于矩阵恢复，我们为线性高光谱解混问题设计了近端梯度算法和自适应动量算法。当高光谱图像的规模很大时，我们引入随机方法降维，提出了随机近端梯度算法。这些结果不仅能为求解大规模矩阵优化问题提供新模型和新算法，而且也可为最优化、信息科学、数据科学、计算机科学技术的交叉融合提供新元素，具有重要的科学意义和实用价值。