MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
MARCH extends recurrent memory via content routing and periodic state checkpoints, improving long-range dependency modeling.
Key Findings
Methodology
This paper introduces MARCH architecture, combining periodic storage of cumulative recurrent states with content-conditioned routing. It periodically checkpoints the evolving state into a bank of anchors, each associated with learned anchor keys for content matching. During token processing, it generates anchor queries to attend over these stored states, aggregating relevant historical information via attention mechanisms. This design enables scalable long-term memory without altering the core recurrence, effectively expanding memory capacity and capturing long-distance dependencies. The approach integrates residual fusion to combine current and historical states, trained end-to-end with standard language modeling objectives.
Key Results
- MARCH outperforms linear attention variants on commonsense reasoning benchmarks, achieving an average accuracy of 41.5%, surpassing baseline models by 1.4%. On OpenBookQA, it improves accuracy by 2.8 percentage points. In long-context retrieval tasks at 16K-32K sequence lengths, MARCH maintains high accuracy, significantly outperforming standard Transformer and Log-Linear Gated-DeltaNet, demonstrating superior long-range memory capabilities.
- On LongBench, MARCH achieves over 25% relative improvement in multi-document QA and summarization tasks compared to Gated DeltaNet. It exhibits robust zero-shot extrapolation to 32K context length, maintaining high accuracy where baseline models fail. Ablation studies confirm that content routing and sparse top-K selection reduce computational costs while preserving performance.
- These results validate MARCH’s effectiveness in long sequence modeling, showing its ability to balance efficiency and memory richness, with broad applicability across NLP tasks requiring extended context understanding.
Significance
This work addresses the fundamental challenge of long-range dependency modeling in neural networks. By introducing content-driven memory expansion, it overcomes the fixed-state bottleneck of linear recurrent models, enabling scalable long-term memory without prohibitive computational costs. The architecture offers a promising solution for applications like long document comprehension, knowledge retrieval, and multi-modal integration, bridging the gap between efficiency and effectiveness in long-sequence NLP. Its innovative storage and retrieval mechanisms set a new direction for scalable, content-aware long-range modeling, with significant implications for both academia and industry.
Technical Contribution
The core technical innovation lies in combining periodic state checkpointing with content-conditioned routing, allowing the model to dynamically access relevant historical states. Unlike prior fixed or multi-version state models, MARCH’s content routing enables selective, content-based retrieval, significantly enhancing long-distance dependency capture. The residual fusion preserves the native recurrence path, facilitating end-to-end training. This approach provides a scalable, flexible memory extension mechanism that maintains computational efficiency, opening new engineering possibilities for large-scale, long-sequence language models.
Novelty
This is the first work to integrate content-based routing with recurrent state checkpointing, creating a scalable long-term memory system that dynamically retrieves relevant historical states based on content similarity. Unlike previous fixed or hierarchical memory approaches, MARCH’s content routing allows for a unified, content-driven memory expansion that adapts to increasing sequence lengths without increasing model parameters. This innovation fundamentally enhances the ability of recurrent models to handle ultra-long sequences, setting a new paradigm in long-range sequence modeling.
Limitations
- Despite its scalability, MARCH still faces computational challenges as the number of stored anchors increases, especially in extremely long sequences. Efficient routing and sparse selection strategies are necessary to mitigate this.
- The frequency of checkpointing and routing parameters require careful tuning; overly sparse checkpoints may miss relevant information, while too frequent checkpoints increase computational overhead.
Future Work
Future research will explore adaptive anchor placement strategies, possibly leveraging reinforcement learning to optimize checkpoint frequency dynamically. Combining sparse routing with hierarchical or multi-scale approaches could further reduce costs. Extending content routing to multi-modal data and integrating it with large pre-trained models are promising directions. Additionally, developing theoretical guarantees for retrieval accuracy and scalability will be crucial for deploying MARCH in real-world systems.
AI Executive Summary
Long-range dependency modeling remains a core challenge in natural language processing. Traditional Transformer models, while powerful, suffer from quadratic complexity, limiting their scalability to very long sequences. Linear attention variants like LinFormer and Performer reduce this complexity but at the expense of capturing distant dependencies effectively. Recurrent models such as Gated DeltaNet offer efficiency but are constrained by fixed state sizes, leading to information loss over long sequences.
This paper introduces MARCH, a novel architecture that combines periodic state checkpointing with content-conditioned routing. By periodically storing cumulative states as anchors and associating each with learned keys, MARCH enables content-based retrieval of relevant historical information. During token processing, it generates queries to attend over these anchors, aggregating pertinent long-term memory via attention mechanisms. This design allows the model to dynamically access and utilize a growing memory bank, effectively extending the capacity of recurrent models without increasing their core complexity.
Empirical results demonstrate MARCH’s superiority across multiple benchmarks. In zero-shot commonsense reasoning, it achieves an average accuracy of 41.5%, outperforming existing linear attention models and matching or exceeding full Transformer baselines. In long-context retrieval tasks at 16K-32K sequence lengths, MARCH maintains high accuracy, significantly outperforming traditional models that fail at these scales. The architecture also excels in multi-document QA and summarization, with performance gains of over 25% relative to strong baselines.
These advances highlight MARCH’s potential to revolutionize long-sequence modeling, enabling scalable, content-aware memory systems suitable for real-world applications like legal document analysis, scientific research, and knowledge-intensive AI systems. While promising, challenges remain in optimizing anchor placement and routing efficiency for ultra-long sequences. Future work will focus on adaptive checkpointing, hierarchical routing, and multi-modal extensions, aiming to further enhance scalability and robustness. Overall, MARCH offers a compelling pathway toward truly long-range, efficient, and intelligent language understanding systems.
Deep Dive
Abstract
Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic computation complexity during training and a key--value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension, while maintaining computational efficiency over long-sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention-style aggregation over all historical anchors along the current state. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computation path.