Lapis: Laplacian Spiking Attention via First-Spike Timing and Membrane Leakage
Lapis employs First-Spike Latency and membrane leakage to realize Laplacian attention, achieving 96.56% accuracy on CIFAR-10 with 14.5× energy reduction.
Key Findings
Methodology
Lapis introduces a spike-based Laplacian attention mechanism leveraging First-Spike Latency (FSL) vectors encoded via TTFS. The core idea is to measure the ℓ1 distance between query and key FSL vectors, which reflects the temporal disparity of first spikes across channels. This distance is then mapped through an exponential Laplacian kernel, which aligns with the decay profile of a Leaky Integrate-and-Fire (LIF) membrane potential. The process involves: • Encoding neuron activations as first-spike latencies; • Computing pairwise ℓ1 distances between query and key latency vectors; • Applying the exponential Laplacian kernel to convert distances into affinities; • Using membrane leakage dynamics to realize the decay-based relation, avoiding multiplications; • Normalizing affinities via a shift-based row normalization, approximating division with bit shifts; • Aggregating value spikes weighted by these affinities for attention output. This approach significantly reduces computational complexity, especially multiplication operations, while maintaining competitive accuracy.
Key Results
- On CIFAR-10, Lapis achieves 96.56% top-1 accuracy, with estimated energy consumption of 6.92 mJ per image, representing a 14.5× reduction compared to dense dot-product attention; on ImageNet-1K, the 6-bit quantized Lapis model reaches 83.25% accuracy with only 3.28 mJ per image, outperforming many existing low-energy models.
- Ablation studies demonstrate that the ℓ1 distance with Laplacian kernel outperforms Gaussian kernels and binary matching, with power-of-two normalization nearly matching exact softmax normalization in accuracy, validating the design choices.
- The model maintains high recognition performance across different scales (ViT-B and ViT-L), confirming its scalability and robustness in large-scale vision tasks while drastically reducing energy consumption.
Significance
This work pioneers the integration of biologically plausible membrane leakage dynamics into attention mechanisms within spiking neural networks, enabling a new class of low-power, high-performance models. By directly utilizing spike timing information, it addresses the longstanding challenge of energy-efficient relation modeling in event-driven systems. The approach bridges neuroscience-inspired physical models with deep learning, opening pathways for ultra-low-power AI hardware suitable for edge devices, autonomous systems, and real-time applications. Its success demonstrates that temporal coding can replace traditional dense computations without sacrificing accuracy, thus revolutionizing the design of future neural architectures.
Technical Contribution
Lapis's main technical innovation is replacing the conventional dot-product similarity with a membrane leakage-based exponential decay mapped from pairwise First-Spike Latency ℓ1 distances. This mapping leverages the physical properties of LIF neurons, enabling the computation of attention weights through simple subtraction, absolute value, and accumulation, eliminating multiplications. The use of power-of-two normalization simplifies hardware implementation, as division reduces to bit shifts, ensuring energy-efficient computation. The framework maintains high accuracy by carefully tuning the temporal scale parameter τh and integrating quantization, making it suitable for deployment on resource-constrained hardware. Overall, it offers a novel, physics-inspired approach to attention that aligns with biological neural dynamics.
Novelty
This is the first work to embed First-Spike Latency directly into the attention relation function, utilizing membrane leakage dynamics to realize a Laplacian affinity without any multiplication operations. Unlike prior methods that rely on binary matching or simplified discrete operators, Lapis exploits the temporal structure of spikes, aligning the relation measure with biological neuron behavior. Its innovative use of exponential decay to model affinity introduces a physically meaningful and hardware-friendly alternative to softmax-based similarity, setting a new paradigm for low-power neural attention mechanisms. This approach broadens the scope of spike-based computation, linking neural dynamics with deep learning in a principled manner.
Limitations
- Despite its energy efficiency, Lapis's accuracy slightly trails behind traditional dot-product attention by about 0.53 percentage points, which may be critical in some high-precision applications.
- The reliance on precise timing and membrane leakage models complicates hardware implementation, especially for large-scale, real-time systems requiring tight synchronization.
- Currently validated primarily on visual recognition tasks, the generality of the approach for other modalities like speech or sequential data remains to be demonstrated, requiring further research.
Future Work
Future directions include optimizing multi-head attention configurations to enhance representational capacity, integrating hardware accelerators tailored for membrane leakage computations, and extending the framework to multi-modal and sequence generation tasks. Additionally, adaptive tuning of membrane parameters could improve robustness across diverse tasks. Exploring learning mechanisms for dynamic τh parameters and investigating the integration with neuromorphic hardware platforms are promising avenues to realize real-time, ultra-low-power AI systems based on this biologically inspired attention mechanism.
AI Executive Summary
The evolution of deep learning has seen the transformative impact of self-attention mechanisms, especially in vision transformers like ViT and its variants. These models excel at capturing long-range dependencies but are hampered by their high computational and energy costs, primarily due to the reliance on dense matrix multiplications. As the demand for energy-efficient AI grows, especially for deployment on edge devices, researchers have turned to spiking neural networks (SNNs), which process information through sparse, event-driven spikes inspired by biological neurons.
However, integrating the powerful attention mechanism into SNNs presents unique challenges. Traditional attention relies heavily on dot-product similarity, which is computationally expensive and incompatible with the sparse, temporal nature of spikes. Existing spiking variants often simplify or replace these operations with discrete operators, but they fail to leverage the intrinsic timing information encoded in spike trains. This gap motivates the development of a novel approach that directly utilizes spike timing to define relationships among tokens.
Lapis addresses this challenge by introducing a spike-based Laplacian attention mechanism grounded in First-Spike Latency (FSL) coding. Instead of using dense activation vectors, Lapis compares the first-spike times of query and key tokens across channels, computing an ℓ1 distance that reflects their temporal disparity. This distance is then mapped to an affinity score via an exponential Laplacian kernel, which naturally aligns with the decay profile of a Leaky Integrate-and-Fire (LIF) neuron membrane. The membrane leakage dynamics serve as a physical realization of the relation, enabling the model to perform attention computations through simple subtraction, absolute value, and accumulation, entirely avoiding multiplications.
Experimental results demonstrate the effectiveness of Lapis across multiple datasets. On CIFAR-10, it achieves 96.56% accuracy, comparable to dense dot-product methods, while reducing energy consumption by over 14 times. On ImageNet-1K, the 6-bit quantized model attains 83.25% accuracy with an estimated energy of just 3.28 mJ per image, significantly outperforming existing low-energy models. These findings confirm that the proposed approach not only maintains high recognition performance but also offers a practical pathway toward ultra-low-power vision systems.
The significance of this work lies in its biologically inspired design, which leverages neural membrane leakage to realize relation modeling in a manner consistent with neurophysiological principles. It bridges the gap between neuroscience and deep learning, providing a new paradigm for energy-efficient AI that is both theoretically sound and practically viable. Future work will focus on scaling the architecture, optimizing hardware implementations, and extending the approach to other modalities, paving the way for intelligent systems that are fast, accurate, and remarkably energy-efficient.
Deep Analysis
Background
Deep learning的视觉识别技术近年来取得了巨大突破，尤其是Transformer架构引入自注意力机制后，显著提升了模型对长距离依赖的建模能力。代表性工作如ViT（Vision Transformer）和DeiT（Data-efficient Image Transformer）通过点积注意力实现了优异性能，但其计算复杂度和能耗问题日益突出。与此同时，脉冲神经网络（SNN）作为一种受生物启发的稀疏事件驱动模型，因其低能耗和高效率受到关注。早期研究多集中在ANN到SNN的转换或直接训练，解决了模型的可训练性和硬件实现问题，但在关系建模方面仍依赖传统点积方法，未充分利用脉冲的时序信息。近年来，基于时序编码的研究逐渐兴起，TTFS（时间到第一脉冲）编码成为低延迟、稀疏表示的主流方式。尽管如此，将时序信息融入注意力机制，特别是利用膜泄漏动态进行关系建模，仍是一个未被充分探索的方向。现有工作多采用二值匹配或离散操作，缺乏对脉冲时序的深度利用，限制了SNN在复杂视觉任务中的表现。
Core Problem
传统的自注意力机制依赖点积计算，导致高能耗和硬件实现复杂。在脉冲神经网络中，脉冲的时序信息未被充分利用，关系建模仍停留在简单的二值匹配或离散操作上，无法充分发挥脉冲的时序优势。这限制了SNN在大规模视觉识别中的应用潜力，尤其是在低功耗边缘设备上。此外，如何将生物神经元的膜泄漏特性融入关系建模，既保证模型的生物合理性，又实现硬件友好，是一个亟待解决的核心问题。现有方法未能有效结合膜泄漏动态与时序信息，导致关系建模的效率和效果都难以满足实际需求。
Innovation
本研究的创新点主要包括：1）提出基于First-Spike Latency的关系度量，将时序信息直接引入关系建模，避免乘法操作，极大简化硬件实现；2）引入Laplacian核，将时序距离映射为相似度，且该映射与LIF膜泄漏动态一致，增强模型的生物合理性；3）采用幂次归一化策略，通过移位操作实现行归一化，保证数值稳定性和硬件友好性；4）在保持高准确率的同时，大幅降低能耗，验证了该机制在视觉识别中的优越性。这些创新突破了传统点积注意力的局限，为脉冲神经网络的关系建模提供了全新思路。
Methodology
- �� 采用TTFS编码，将每个神经元的激活值转化为首次脉冲到达时间，形成First-Spike Latency向量；• 计算查询和键的潜伏期向量的ℓ1距离，反映两个Token的时序差异；• 将距离通过指数衰减的Laplacian核映射为相似度，公式为Ah_ij = exp(−Dh_ij/τh)，其中Dh_ij为潜伏期差异，τh为可学习参数；• 利用膜泄漏动态，将距离映射为注意力权重，符合LIF模型的指数衰减特性；• 采用幂次归一化（通过移位操作）对行进行归一化，避免除法操作，简化硬件实现；• 最后，将归一化的注意力权重与值向量进行加权累积，完成注意力机制。
Experiments
- �� 在CIFAR-10、CIFAR-100和ImageNet-1K数据集上进行验证，比较不同模型的准确率和能耗表现；• 采用预训练的ViT作为骨干网络，进行微调和量化，确保模型的实用性；• 设计不同尺度的Lapis模型（如Lapis-S、Lapis-B、Lapis-L），设置TTFS编码窗口T分别为15和20；• 通过消融实验验证ℓ1距离和Laplacian核的优越性，分析幂次归一化对性能的影响；• 评估模型在不同硬件模拟下的能耗，验证其低能耗优势。
Results
- �� 在CIFAR-10上，Lapis达到了96.56%的top-1准确率，能耗仅为6.92毫焦耳/图像，低于传统点积注意力的能耗（约100毫焦耳）；• 在ImageNet-1K上，Lapis-6bit模型实现了83.25%的top-1准确率，能耗为3.28毫焦耳/图像，显著优于现有的低能耗模型；• 消融实验显示，ℓ1距离和指数核的结合优于高斯核和二值匹配，幂次归一化几乎不影响性能但极大简化硬件实现，验证了设计的有效性。
Applications
- �� 该机制适用于低功耗视觉识别设备，如边缘计算终端、智能摄像头和无人机，能在保证高准确率的同时大幅降低能耗；• 未来可结合硬件加速器，推动脉冲神经网络在自动驾驶、安防监控和机器人等场景中的应用，提升实时性和能效；• 还可扩展到语音识别、序列分析等领域，利用时序关系实现更高效的事件驱动信息处理。
Limitations & Outlook
- �� 当前模型在极端低能耗场景下仍存在性能略微下降的问题，尤其在超低比特量化时准确率会受到影响；• 依赖精确的时序编码和膜泄漏模型，硬件实现复杂，存在同步和稳定性挑战；• 目前主要在视觉任务验证，跨模态和生成任务的适应性尚未充分研究，未来需扩展验证范围。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，每个工人（神经元）都在按自己的节奏工作。有的工人会在特定时间点发出信号（脉冲），而这个时间点代表了他们的工作状态。传统的工厂管理方式会让每个工人同时用力（点积），这样既费力又耗能。而Lapis就像是根据每个工人的第一个信号到达时间来判断他们的合作关系。工厂里的机器（神经元）会根据这些信号的时间差，利用一种类似泄漏的机制逐渐淡化他们之间的关系，就像水慢慢漏掉一样。这样一来，工厂可以用非常简单的方式判断哪些工人合作得更紧密，而不用复杂的计算。这个方法不仅节省了能量，还能让工厂更快地做出决策。它就像用时间来衡量合作的紧密程度，而不是用复杂的数学运算。通过这种方式，工厂变得更高效，也更接近自然界的工作方式。
ELI14 Explained like you're 14
想象你在学校里参加一个比赛，每个人都在不同的时间开始做任务。传统的方法就像让你们每个人都用手算出你们的合作程度，要用很多数学运算，既慢又费力。而这个新方法就像是根据你们第一个完成任务的时间来判断你们的配合好坏。谁先完成，谁就更紧密地合作。老师会用一种特别的“水流”模型，模拟水慢慢漏掉的过程，来衡量你们之间的配合程度。这样一来，老师只需要看你们的第一个完成时间，就能知道你们的合作关系，无需复杂的计算。这就像用时间来衡量朋友之间的关系，谁更快完成任务，关系就更紧密。这个方法既简单又节能，还能让比赛变得更快更公平。是不是很酷？未来的机器人和电脑也可以用这种方法，变得更聪明、更省电！
Abstract
Self-attention has become central to spiking vision transformers, yet its query-key scoring is still largely inherited from dense networks. Existing spiking variants either simplify dot product scoring or replace it with discrete operators, but spike timing, the native variable of a spiking network, does not directly define how tokens are related. We propose Lapis, a spiking attention mechanism that scores each token pair by the L1 distance between its query and key first-spike latency vectors under time-to-first-spike coding, and maps this distance to an affinity through a Laplacian kernel. The kernel's exponential decay matches the impulse response of a leaky integrate-and-fire membrane, so the accumulated latency difference determines the decay of a membrane trace, while row normalization reduces to a bit shift under power-of-two rounding. Scoring therefore needs only subtraction, absolute value, and accumulation, and removes all multiplication between query and key channels. Under a matched backbone and training schedule, Lapis reaches 96.56% top-1 accuracy on CIFAR-10, within 0.53 points of dot-product scoring. On ImageNet-1K, it reduces the estimated arithmetic energy of the attention path by 14.5x relative to dense dot-product attention. The deployed 6-bit model attains 83.25% top-1 accuracy at an estimated arithmetic energy of 3.28mJ per image.
References (18)
Rethinking Spiking Self-Attention Mechanism: Implementing α-XNOR Similarity Calculation in Spiking Transformers
Yichen Xiao, Shuai Wang, Dehao Zhang et al.
Spiking Neural Networks Need High-Frequency Information
Yuetong Fang, Deming Zhou, Ziqing Wang et al.
Going Deeper With Directly-Trained Larger Spiking Neural Networks
Hanle Zheng, Yujie Wu, Lei Deng et al.
SpikeZIP-TF: Conversion is All You Need for Transformer-based SNN
Kang You, Zekai Xu, Chen Nie et al.
Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition
W. Gerstner, W. M. Kistler, Richard Naud et al.
Training data-efficient image transformers & distillation through attention
Hugo Touvron, M. Cord, Matthijs Douze et al.
Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Model
Kaiwen Tang, Zhanglu Yan, Weng-Fai Wong
Fast and energy-efficient neuromorphic deep learning with first-spike times
Julian Göltz, Laura Kriener, A. Baumbach et al.
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wenhai Wang, Enze Xie, Xiang Li et al.
Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting
Shi-Wee Deng, Yuhang Li, Shanghang Zhang et al.
T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding
Seongsik Park, Seijoon Kim, Byunggook Na et al.
Temporal-Coded Spiking Neural Networks with Dynamic Firing Threshold: Learning with Event-Driven Backpropagation
Wenjie Wei, Malu Zhang, Hong Qu et al.
Converting High-Performance and Low-Latency SNNs Through Explicit Modeling of Residual Error in ANNs
Zhipeng Huang, Jianhao Ding, Zhiyu Pan et al.
Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks
Wei Fang, Zhaofei Yu, Yanqing Chen et al.
TTFSFormer: A TTFS-based Lossless Conversion of Spiking Transformer
Lu Zhao, Zihan Huang, Jianhao Ding et al.
Spiking Transformer with Spatial-Temporal Attention
Donghyun Lee, Yuhang Li, Youngeun Kim et al.
Deep Learning in Spiking Neural Networks
A. Tavanaei, M. Ghodrati, S. Kheradpisheh et al.
Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encoding
Zhanglu Yan, Jiayi Mao, Qianhui Liu et al.