LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression

The evolution of multimodal AI has seen large vision-language models (LVLMs) like Florence, LLaVA, and Flamingo achieving impressive feats in visual understanding and reasoning. These models leverage large-scale pretraining on image-text pairs, enabling tasks such as visual question answering, captioning, and complex reasoning. Early efforts focused on feature fusion and supervised fine-tuning, which improved multimodal alignment. However, as models scaled up, long-horizon reasoning revealed a critical flaw: visual attention diminishes over extended generation, leading to a phenomenon called visual forgetting. This results in models relying increasingly on language priors, causing hallucinations and reduced accuracy. Recent approaches incorporated reinforcement learning (RL), such as DeepSeek and VL-Rethinker, to optimize reasoning trajectories via reward signals. Despite progress, these methods do not explicitly regulate attention dynamics, especially early-stage grounding and sink token over-concentration, leaving a gap in robustness and interpretability.

The core challenge is that during long reasoning sequences, LVLMs exhibit a decline in visual attention, impairing their ability to ground responses in visual evidence. This manifests as early attention decay—where initial visual cues are lost—and sink token over-concentration—where attention collapses onto uninformative tokens, reducing the model’s focus on relevant regions. These issues cause errors to propagate autoregressively, degrading performance significantly. Addressing this requires a mechanism to maintain stable, task-relevant visual attention throughout the reasoning process, both temporally and spatially. Existing solutions lack a unified, training-based approach that dynamically adjusts attention, making models vulnerable to attention collapse and hallucinations in complex scenarios.

LASER introduces a dual-reward system integrated into the RL training framework. The Visual Grounding Reward (Rvis) encourages the model to sustain high attention on semantically salient visual tokens, especially in early reasoning stages, by monitoring attention proportions over non-sink tokens. The Sink Suppression Reward (Rsupp) penalizes excessive attention on sink tokens—those that absorb disproportionate attention without semantic relevance—by identifying them through activation pattern analysis. This combination ensures early visual evidence is preserved and attention is redistributed away from uninformative regions. Unlike prior post-hoc or heuristic methods, LASER embeds dynamic attention regulation directly into training, offering a principled, scalable solution that improves both grounding stability and interpretability.

• �� Build on the Group Relative Policy Optimization (GRPO) framework, which compares rewards within response groups to optimize policy.
• �� Identify sink tokens automatically based on hidden state activation patterns, using a threshold on activation magnitudes.
• �� Design Visual Grounding Reward (Rvis) to monitor the proportion of attention on non-sink tokens, penalizing decay below a peak level.
• �� Develop Sink Suppression Reward (Rsupp) to penalize disproportionate attention on sink tokens, measured by the ratio of sink to total attention.
• �� During training, combine these rewards with accuracy signals, adjusting model parameters to maintain early visual grounding and suppress sink attention.
• �� Fine-tune hyperparameters to balance the rewards, ensuring stable attention trajectories across extended sequences.

Using the Qwen-2.5-VL-7B model, trained on 45K RL samples across diverse tasks, including math reasoning, logic, and perception, the authors evaluate LASER against multiple baselines like GPT5 and Gemini-2.5-Pro. Experiments involve ablation studies to isolate reward effects, attention trajectory analysis, and performance metrics such as accuracy improvements. The training employs hyperparameter tuning for reward weights, with evaluation on 8 benchmarks. Results show consistent accuracy gains, especially in complex tasks, with attention maps confirming sustained visual grounding. Ablation confirms both rewards are necessary for optimal performance, and analysis of attention dynamics demonstrates the effectiveness of the regulation mechanism.

LASER achieves a 1.9% accuracy increase on MMStar and a 5.7% boost on visual perception tasks, outperforming prior methods. Attention trajectory plots reveal models trained with LASER maintain stable focus on visual evidence throughout reasoning, reducing drift and collapse. Ablation results show removing either reward diminishes gains, highlighting their combined importance. The method effectively mitigates early attention decay and sink token over-concentration, leading to more accurate and interpretable reasoning outputs.

This approach can be applied in autonomous vehicles, robotic perception, and multimodal question answering systems, where maintaining visual grounding is critical. It requires pre-trained models and reinforcement learning training pipelines, suitable for scenarios demanding high reliability and interpretability. Long-term, LASER’s principles could inform training strategies for other sequential decision-making models, enhancing robustness in real-world deployments.

The method depends on computationally intensive RL training and manual hyperparameter tuning, which may limit scalability. Its effectiveness in highly noisy or ambiguous environments remains to be tested. Additionally, in extremely long or complex sequences, some attention drift may still occur, indicating the need for further refinement of the regulation mechanism.