面向神经形态系统的离散态深度网络在线学习理论

Recently the booming breakthroughs achieved by deep learning in various applications totally open the AI Era. In the meanwhile, neuromorphic chips and systems based on Deep Neural Networks (DNNs) models are developing explosively, and are promising to support wide fields in the near future, including smart phones, robotics, unmanned aerial vehicles, intelligent drive, wearable devices, security monitoring, etc. On the other side, theories on hardware friendly DNNs are proposed, and have drawn great attention. However, current discrete DNNs have many disadvantages which limit their applications, such as only supporting offline use, memory hungry, and limited and inflexible discrete states. Accordingly, this project will carefully investigate the neuromorphic hardware constrains, and build the online learning model of discrete DNNs. Then, we expand the discrete space of parameters and states to multi levels and make it flexible to modify, and study on chain rule of derivative and theory of gradient descent, prove its convergence, and propose back propagation (BP) learning algorithm to enable the direct training of discrete DNNs. Furthermore, we will form the guidance theory for future online neuromorphic platform, and design efficient neuromorphic architecture for online learning, build the corresponding software and hardware platform, and finally demonstrate various tasks. This project will drive the development of future online neuromorphic system and provide guidance theory, promote the co-development of neuromorphic system and DNNs theory, help China get a head start in future brain-inspired field of online learning, and strengthen our competitiveness in the coming AI Era.

近年来深度学习在各类应用中接连突破，强势拉开了智能化时代的革命序幕。在此背景之下，基于深度神经网络(DNNs)的神经形态系统成为研究热点，未来将广泛应用于智能手机、机器人、无人机、智能驾驶、可穿戴设备和安防监控等领域；与此同时，硬件友好的低精度离散态DNNs理论也迅速萌芽，并引起国内外学者的广泛关注。本项目针对目前离散态DNNs理论仅能离线使用、消耗大量片外高精度存储资源的缺陷，深入研究神经形态系统的硬件约束条件，建立离散态DNNs在线学习问题的优化模型。据此，研究多阶离散态参数空间和状态空间的DNNs链式求导法则，提出基于概率迁跃误差反向传播算法的思路；凝练总结神经形态平台在线学习计算架构的设计理论，并完成理论算法的演示验证。本项目完成后，有望促进神经形态系统和深度神经网络理论的协同发展，为未来基于神经形态在线学习类脑计算系统的发展做出贡献。

神经形态计算是一种有望实现低功耗类脑智能的方案，基于二进制脉冲的通信方式是其实现低功耗的关键。神经形态系统的二进制硬件约束可以转化为离散态深度神经网络（DNNs）在线学习的优化问题。本项目深入研究了离散态DNNs的优化问题，首次提出了块动态等距理论，通过度量DNNs中梯度范数的变化，从理论上分析了神经网络梯度消失或爆炸的原因。在此基础上，改进了一些现有的深度学习训练方法，包括激活函数选择、参数初始化、批标准化等策略。根据所提出的块动态等距理论，并结合神经形态计算中脉冲神经元的二进制激活特性，提出了一种新的归一化技术，不仅在速度上远远超过传统方法，并且没有精度损失。在硬件部署时，针对目前神经形态系统中实现批归一化成本过高的问题，通过设计“批次采样”和“特征采样”两种方法，并结合多路策略减少数据相关性，极大地降低了在硬件上执行批归一化所需的计算资源。同时，为了进一步降低在线学习芯片在训练期间所需的计算和存储资源，通过分解计算步骤并融合多种量化函数，建立了全流程8-bit定点数在线学习框架。此外，针对当前大规模神经形态算法训练过慢的问题，通过建立典型脉冲神经元的显式迭代版本，提出了一种基于Pytorch的大规模脉冲神经网络训练方法，将训练网络所需的时间缩短了十倍以上，并在多种数据集上取得了当前最优的结果。最后，基于高性能仿真计算平台，广泛收集神经形态芯片的测量数据，开发了一种用于大规模网络硬件仿真的映射编译器和循环精度模拟器，用于算法的测试和验证。本项目为神经形态算法的设计及硬件部署，大规模网络加速训练等提供了理论基础。受到所提出的理论启发，本项目提出了一系列方法来提升神经形态系统的性能，为神经形态在线学习芯片走向资源受限的边缘计算场景奠定了坚实基础。