深度神经网络特性分析与架构优化研究

Residual Network (ResNet) can effectively solve the degradation problem in training Deep Neural Network (DNN), and has achieved great success in applications. However, theoretical studies on properties and mechanisms of DNN has received much less attention due to difficulties in analysis. Problems of DNN are often delth with engineering experiences, due to lack of effective analytic methods. In this projects, we study some properties of DNN from theoretical viewpoint, through analyzing those of ResNet; the theoretical analysis includes: relationship between ResNet and feedforward Neural Network, how gradients are backward propagated, instabilities therein, the convexity of the objective function, local minima and saddle points, and how image features are extracted. Based on the theoretical analysis, we propose a method for optimizing the architecture of ResNet based DNN，which can be used to improve the performance，compress the model size and reduce computational complexity. The rationality and effectiveness of the analysis method and the optimization method will be evaluated by using image recognition and face recognition databases.

残差网络（Residual Network，ResNet）能有效解决深度神经网络（Deep Neural Network，DNN）训练退化问题，在应用上取得了很大成功。但对于DNN的特性、机理的理论研究，由于难度大，鲜有展开；对于DNN的诸多问题，由于缺乏有效的理论分析体系，因而只靠工程经验去艰难解决。本项目从ResNet入手， 对DNN的特性做出理论分析，包括：ResNet与前馈型神经网络的关系、梯度传播方式、梯度不稳定问题、鞍点与极值点的特点、目标函数的凸性、以及ResNet提取图像特征的特点等问题；在此理论分析基础上，提出基于ResNet的架构优化方法（Architecture Optimized ResNet, AO-ResNet），该方法可以裁剪网络模型架构参数，同时提升DNN模型性能，减少内存、运算开销。拟采用图像识别和人脸识别数据验证本项目理论分析和优化方法的合理性与有效性。

在深度学习方法席卷计算机视觉各领域的背景下，本项目针对深度学习方法中的网络架构设计和优化方法展开研究，提出新方法和新理论提升深度学习方法的有效性。在网络架构方面，提出融入注意力机制的深度学习网络，实现融合全局和局部特征的混合特征网络设计，提出上下文敏感注意力网络架构；在优化方法方面，提出排外和一直约束项引导的网络训练方法，实现自适应训练样本采样和图卷积网络的优化方法。将上述方法在人脸识别，人脸检测，目标检测，行人重识别等多种计算机视觉任务上进行了验证，提升了相关任务的分析识别准确性，体现了所提出深度学习方法的先进性，对图像相关领域的模式分析识别有一定的借鉴作用。