Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Proposes SATS and Token Routing to improve activation sparsification in LLMs, enhancing efficiency and performance.
Key Findings
Methodology
This paper introduces SATS, which replaces percentile-based threshold calibration with sensitivity-aware layerwise thresholds using local activation error proxies. It dynamically adjusts each layer’s gate thresholds to control output distortion. Additionally, a lightweight Token Routing mechanism is proposed, which, based on token identity, chooses between a dense baseline path and a sparse path during inference. Experiments on LLaMA 3.1 8B and Qwen 3 8B demonstrate that SATS outperforms traditional calibration at matched sparsity, and Token Routing further improves the quality-throughput trade-off by selectively routing tokens, maintaining speed while boosting accuracy.
Key Results
- At target sparsity levels of 30% and 50%, SATS achieves approximately 0.5-1% higher average accuracy and lower perplexity on Wiki-Text2 and RefinedWeb datasets compared to percentile-based methods. For example, with a 50% sparsity target, SATS reaches a realized sparsity of 43.9%, outperforming traditional methods calibrated at 75%. Token Routing at 50% routing target improves average task accuracy by 1-2%, especially in fine-tuned models, while preserving inference speed. These results highlight the method’s robustness across multiple tasks and datasets.
- Across downstream tasks, SATS and Token Routing consistently outperform baseline sparse methods, demonstrating improved effective sparsity and inference efficiency. The experiments validate that the combined approach effectively balances model performance and speed, with notable gains at higher sparsity levels.
Significance
This work advances the state-of-the-art in activation sparsification by incorporating sensitivity-aware calibration, addressing the limitations of percentile-based thresholds. The dynamic Token Routing enables fine-grained token-level adaptation, significantly boosting inference efficiency without sacrificing accuracy. These innovations are critical for deploying large models in resource-constrained environments, reducing computational costs, and enabling real-time applications. The approach bridges the gap between model compression and performance retention, offering a practical pathway for scalable AI deployment.
Technical Contribution
The core technical contribution is the sensitivity-aware threshold calibration (SATS), which estimates local output distortion to set layerwise gate thresholds adaptively. This replaces static percentile calibration, reducing information loss. The second key contribution is the token-level routing mechanism, which uses a precomputed lookup table of token risks to dynamically route tokens during inference, minimizing overhead. These methods are compatible with existing activation sparsification frameworks, providing a scalable, effective solution for large models. Extensive experiments validate their superiority over prior methods.
Novelty
This is the first work to integrate sensitivity-aware threshold calibration with token-level dynamic routing in large language models. Unlike prior approaches relying solely on activation percentiles, SATS considers local output distortion, enabling more precise sparsification. The token routing mechanism introduces a novel, low-overhead way to adapt inference paths based on token context, offering a new dimension of model efficiency. Together, these innovations push the frontier of model compression and acceleration, especially for large-scale transformers.
Limitations
- The methods perform less effectively at extremely high sparsity levels (>70%), where performance degradation becomes significant. The sensitivity proxy depends on calibration data quality; distribution shifts can impair calibration accuracy. Token routing, while low overhead, may still introduce minor latency, especially in very sparse regimes. Further, the approach requires additional calibration steps, which may complicate deployment pipelines. Future work should focus on adaptive calibration and more robust routing strategies to handle diverse scenarios.
Future Work
Future research will explore adaptive, self-supervised calibration techniques that do not rely heavily on calibration datasets. Extending token routing to multi-task and multi-modal models can further enhance efficiency. Hardware-aware optimization, integrating with specialized accelerators, will be crucial for real-world deployment. Additionally, developing theoretical guarantees for the robustness of sensitivity-based thresholds and routing decisions remains an open avenue.
AI Executive Summary
Large Language Models (LLMs) have revolutionized natural language processing, yet their enormous computational demands hinder widespread deployment. Traditional sparsification techniques, such as thresholding activations based on percentiles, often ignore the varying importance of neurons and tokens, leading to suboptimal trade-offs between speed and accuracy. To address this, the present study introduces two key innovations: Sensitivity-Aware Thresholding for Sparsity (SATS) and a lightweight Token Routing mechanism.
SATS replaces static percentile-based calibration with a dynamic, sensitivity-guided method that estimates local output distortion caused by neuron zeroing. By setting per-layer thresholds that respect a global error budget, SATS achieves higher effective sparsity while preserving model performance. Meanwhile, Token Routing leverages token-level risk estimates, stored in a lookup table, to decide during inference whether a token should pass through the dense baseline path or a sparse, thresholded path. This fine-grained control allows the model to adaptively balance speed and accuracy.
Experimental results on LLaMA 3.1 8B and Qwen 3 8B demonstrate that SATS outperforms traditional percentile calibration at the same sparsity levels, improving average accuracy by approximately 0.5-1%. Token Routing further enhances the quality-throughput trade-off, boosting accuracy by 1-2% while maintaining inference speed comparable to dense models. These findings confirm that sensitivity-aware calibration and token-level adaptation are powerful tools for efficient large-scale inference.
The broader impact of this work lies in enabling resource-efficient deployment of large models, reducing hardware costs, and expanding accessibility. The methods are compatible with existing frameworks and can be integrated into real-world systems, including edge devices and cloud services. Limitations include performance drops at very high sparsity and dependence on calibration data quality. Future directions involve adaptive calibration, hardware-aware optimization, and extending techniques to multi-modal models, promising a new era of scalable, efficient AI.
Deep Dive
Abstract
Efficient inference in Large Language Models (LLMs) requires deciding where computation can be reduced while preserving model quality. We study this problem through multilayer perceptron (MLP) activation sparsification and token-level conditional routing. We first propose Sensitivity-Aware Thresholding for Sparsity (SATS), a threshold calibration method to choose layerwise gate thresholds using a local MLP output sensitivity proxy rather than calibrating thresholds directly from activation percentiles. While SATS retains the existing mechanism of sparsifying MLP activations by thresholding gate activations, it replaces percentile-based calibration with a sensitivity-aware selection rule. We then introduce a lightweight token routing framework that dynamically selects between a base path and a modified path on a per-token basis, rather than applying the modified computation uniformly to all tokens. We evaluate both methods on multiple recent open-weight LLMs. Our results show that SATS improves over the threshold-based sparsification baseline at matched actual sparsity and that token routing yields a more favorable quality-throughput trade-off than static activation modification baselines. Overall, our results suggest that improved threshold calibration and token routing can improve the quality-throughput trade-off in LLMs.