Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer

Spiking Neural Networks have emerged as a promising energy-efficient alternative to Artificial Neural Networks, utilizing event-driven computation and binary spikes for information transfer. Despite their energy efficiency, SNNs face significant challenges in achieving high task accuracy, particularly when integrated with CNN-based architectures. A potential solution is the combination of Transformer models with SNNs. This paper addresses the challenge of adapting the self-attention mechanism of Transformers to the spiking paradigm by introducing a novel approach: Accurate Addition-Only Spiking Self-Attention (A$^2$OS$^2$A). Unlike existing methods that rely exclusively on binary spiking neurons for all components of the self-attention mechanism, our approach incorporates binary, ReLU, and ternary spiking neurons. This hybrid strategy substantially improves accuracy while maintaining non-multiplicative computations. Furthermore, our method eliminates the need for softmax and scaling operations. Extensive experiments demonstrate that the A$^2$OS$^2$A-based Spiking Transformer outperforms existing SNN-based Transformers on both static and neuromorphic datasets, achieving an accuracy of 78.66\% on ImageNet-1K. Our work represents a significant advancement in SNN-based Transformer models, offering a more accurate and efficient solution for real-world applications.