深度神经网络无监督定点量化方法研究

In recent years, deep neural networks (DNNs) have been evolving rapidly and have attracted widespread attention among researchers and developers throughout the world, including both academia and industry. However, as the performance of deep neural networks continues to increase, the network structure also becomes more and more complex. Such high computational and storage cost becomes the major obstacle to the deployment of neural networks. Therefore, network acceleration and compression techniques represented by fixed-point quantization have attracted more and more attention. Most of the current fixed-point quantization methods are based on a large number of labeled samples for supervised training. However, training samples are often difficult to obtain, and the acquisition of labeled data is even more difficult. Under this background, the study of unsupervised network quantization has important theoretical and application values. ​​In this project, we try to study the network quantization based on a few of unlabeled data, and solve the key problems within it. The main contents of our research are as follows. First, we study the automatic network architecture search methods, which are friendly to network quantization. Then, we study the quantization error correction method based on local information, and the structured knowledge distillation method based on global information, for unsupervised network quantization. Last, we will study the unsupervised quantization methods based on adversarial training.

近几年来，深度神经网络发展迅猛，引起了包括学术界和工业界的广泛关注。然而，随着深度神经网络性能的不断提升，网络的计算量和存储也随之增大，并逐步成为阻碍神经网络应用落地的主要障碍。因此，以定点量化为代表的深度神经网络的加速与压缩技术受到了越来越多的关注。然而，目前绝大部分定点量化方法都是基于大量有标签样本进行监督训练，而大量的训练样本往往很难获得，带标签数据的获取更是难上加难。在此背景下，研究神经网络无监督定点量化，具有非常重要的科学意义和实用价值。本项目旨在研究基于少量无标注样本的神经网络定点量化技术，并针对其中的关键科学问题展开深入探讨。主要研究内容包括：研究适用于低比特量化的自动化网络结构搜索，基于局部信息（知识）的量化误差校正的无监督量化，基于全局信息（知识）的结构化知识蒸馏的无监督量化，以及基于对抗学习的无监督量化方法。

本项目围绕深度神经网络加速压缩以及无监督定点量化研究中的关键科学问题进行展开，主要研究了神经网络极低比特定点量化、神经网络混合精度量化、小数据无监督量化、以及软硬协同量化及计算架构设计等内容。在神经网络极低比特量化方面，分析了编码方式对二值及三值量化的影响，提出稀疏二值量化及软阈值三值量化等方法，提升极低比特定点量化效率；在混合精度量化方面，建立任务损失和量化位宽分配之间的联系，提出了一种原则性的框架来解决混合精度量化问题；在小数据无监督量化方面，理论证明了全局量化损失与局部量化损失直接的关联，并提出基于分解以及误差校正的无监督量化方法；在软硬件协同设计方面，提出了硬件友好的量化编码方式以及高效的定点计算架构，提升量化网络的推理效率；最后，本项目搭建了神经网络定点量化验证平台，并集成了本项目提出的量化压缩方法。