大规模非凸正则化机器学习求解算法研究

Non-convex regularizations have great advantages to improve the model's ability to learn and to generalize. However, there is still no algorithm designed to solve the large-scale non-convex and non-smooth machine learning problems with inequality constraints, and heuristic algorithms don't have a theoretical convergence guarantee. To address this issue, this project intends to focus on the research on optimization methods for large-scale non-convex regularized machine learning. Its research contents include: 1) To combine path following method and splitting operator method, and to propose parallel and distributed algorithms for large-scale data-driven applications. 2) To construct an asymptotic approximation framework for the general non-convex non-smooth inequality constrained optimization problems based on the path following method to avoid some of the local optimal solutions that the algorithm may be trapped into. 3) To generate an adaptive selection strategy for the hyperparameters in the model to alleviate the performance jitter in the training process. 4) To propose a high-performance parallel and distributed algorithm for large-scale data-driven applications. Some of the research results previously published in top conferences of operation research, conferences of machine learning, and SCI indexed journals have laid a solid foundation for the research work of this project.

非凸正则项在提升机器学习模型表达能力和泛化能力方面具有潜在优势，但对大规模非凸、非平滑具有不等式约束的机器学习问题，目前还缺少为之设计的求解算法，而启发式求解算法又缺乏收敛性保证。为解决这类问题，本项目开展大规模非凸机器学习求解算法研究，研究内容包括：1)结合路径跟随方法和分裂算子法，面向大规模数据驱动应用场景提出并行和分布式求解算法；2)根据路径跟随方法构造一般非凸非平滑不等式约束问题的渐进逼近框架，规避部分在求解过程中可能遇到的随机局部优化解；3)同时，对模型中的超参数采用自适应选择策略，缓解求解过程中的性能抖动；4)对大规模数据驱动应用，开发高性能并行和分布式求解算法，以满足实际应用问题的需要。部分前期已发表于控制领域和机器学习领域顶级会议以及SCI检索期刊上的研究成果为本项目的顺利展开奠定了坚实基础。

本课题面向大规模非凸正则化机器学习问题，研究并实现高效稳健的求解算法，研究内容主要包括： 1）结合路径跟随方法和分裂算子法，构造一般非凸非平滑不等式约束问题的渐进逼近框架，并分析在渐进逼近框架下求解算法的收敛性质；2）对于大规模数据密集型应用，研究实现并行和分布式求解算法，以应对算法在现实生产生活中的应用需求； 3）将求解算法应用于非凸SVM求解、RNN神经网络训练等实际问题中。将循环神经网络训练问题重新形式化为一个带约束最优化问题，进而使用本课题提出的算法进行训练求解，可快速、稳定的训练一大类RNN神经网络模型，并进一步应用于事件抽取、关系分类、真实世界研究等多个任务之中。