- This event has passed.
INRC Forum: Thomas Nowotny
14 March, 2023 @ 08:00 - 09:00 PDT
Loss shaping enhances exact gradient learning with EventProp in Spiking Neural Networks
Abstract: In a recent paper Wunderlich and Pehle (2021) introduced the EventProp algorithm that enables training spiking neural networks by gradient descent on exact gradients. In this talk I will present extensions of EventProp to support a wider class of loss functions and an implementation in the GPU enhanced neuronal networks framework (GeNN) which exploits sparsity. The GPU acceleration allows us to test EventProp extensively on more challenging learning benchmarks. We find that EventProp performs well on some tasks but for others there are issues where learning is slow or fails entirely. We have discovered that the problems relate to the exact gradient of the loss function not providing information about loss changes due to spike creation or spike deletion. Depending on the details of the task and loss function, descending the exact gradient with EventProp can lead to the deletion of important spikes and so to an inadvertent increase of the loss and decrease of classification accuracy and hence a failure to learn. In other situations, the lack of knowledge about the benefits of creating additional spikes can lead to a lack of gradient flow into earlier layers, slowing down learning. We are trying to overcome these problems in the form of `loss shaping’, where we introduce a suitable weighting function into an integral loss to increase gradient flow from the output layer towards earlier layers. I will show example result for the Spiking Heidelberg Digits and sequential spiking MNIST where we achieve (close to) state-of-the-art performance.
Bio. Prof. Thomas Nowotny has a background in theoretical physics. After his PhD from Leipzig University in 2001 he started working in Computational Neuroscience and bio-inspired AI at the Institute for non-linear Science at UCSD. He is now a Professor in Informatics at the University of Sussex and the head of the AI research group. His interests include olfaction, hybrid systems, spiking neural networks and their efficient simulation, bio-inspired AI and algorithms for neuromorphic computing.
For the meeting link, see the full INRC Forum Spring 2023 Schedule (accessible only to INRC Affiliates and Fully Engaged Members).