前馈神经网络的结构稀疏化设计与分析

On the premise of appropriate learning accuracy, the number of the neurons and weights of a neural network should be as less as possible (constructional sparsification), so as to reduce the cost, and to improve the robustness and the generalization accuracy. This project studies the constructional sparcification of feedforward neural networks by using regularization methods, and it contains the following main points: 1) We propose to punish the redundant neurons to get a more effective nodes sparsification, while the traditional approach punishes the weights to get weight sparcification. 2) Apart from the traditional L1 regularization for sparsification, we also use L1/2 regularization. To remove the oscillation in the iteration process due to the nonsmoothness of the L1/2 regularizer, we propose to smooth it in a neighborhood of the nonsmooth point to get a smoothing L1/2 regularizer. By doing so, we expect to improve the efficiency of the L1/2 regularizer so as to surpass the L1 regularizer. 2) We propose to study the sparsification of the input layer neurons. It not only is a part of the whole network sparsification, but also provides a tool for the sparsification of variables of general nonlinear problems. 3) For the output representation for multi-classification problems, we propose to use the binary approach to replace the traditional one-for-each approach so as to simply and effectively reduce the number of the output neurons.

在保证适当学习精度前提下，神经网络的权值连接以及神经元应该尽可能少（结构稀疏化），从而降低成本，提高稳健性和推广精度。本项目用正则化方法研究前馈神经网络的结构稀疏化，有以下几个要点：1）传统的神经网络正则化通过惩罚冗余权值连接达到权值稀疏化；我们主张通过惩罚冗余单元而达到效率更高的单元稀疏化。2）除了传统的用于稀疏化的L1正则化之外，我们还采用近几年流行的L1/2正则化。为了解决L1/2正则化算子不光滑，容易导致迭代过程震荡这一问题，我们试图在不光滑点的一个小邻域内采用磨光技巧，构造一种光滑化L1/2正则化算子，以期达到比L1正则化更高的稀疏化效率。3）我们首倡研究输入层单元稀疏化，不但作为整个网络结构稀疏化的一部分，更使得神经网络成为非线性压缩感知的一个可行工具。4）用于多分类问题时，我们首倡输出层单元采用二进制方式，代替传统的亮灯方式，简单高效地减少输出单元。

本项目利用正则化方法研究前馈神经网络的结构稀疏化，发表23篇期刊论文。其中SCI论文18篇，EI论文2篇，核心期刊论文3篇。取得以下主要成果：1. 针对BP神经网络的L1/2正则化学习算法，完成7篇论文。2. 用于高阶、区间、模糊、脉冲、ELM神经网络、模糊粗糙等几类特殊的神经网络的学习算法，发表论文9篇。3. 其他智能计算方法，发表论文7篇。