Sessa: Selective State Space Attention

Modern sequence models are predominantly led by Transformers, where self-attention mixes information from the visible context in an input-dependent manner. However, when retrieval is not sharp and attention remains diffuse over an effective support, the influence of any individual token is diluted, especially in full-prefix settings where the influence of old tokens scales as O(1/ℓ). Structured state-space models process sequences recurrently through an explicit feedback path; selective variants like Mamba make this feedback input-dependent, yet when freeze time cannot be sustained over long intervals, their long-range sensitivity decays exponentially with lag. Existing architectures either retrieve from the past in a single read or propagate information through a single feedback chain. We introduce Sessa, a decoder that places attention inside a feedback path, enabling recurrent many-path aggregation within a layer. Under stated assumptions, Sessa admits regimes with a power-law memory tail in lag ℓ of order Θ(ℓ^{-β}), where 0 < β < 1, which is asymptotically slower than 1/ℓ; moreover, this rate is tight in an explicit diffuse uniform-routing setting where the influence is Θ(ℓ^{-β}). Under the same conditions, only Sessa among the compared model classes realizes flexible selective retrieval, including non-decaying profiles. Empirically, under matched architectures and training budgets, Sessa achieves the strongest performance on our long-context benchmarks while remaining competitive with Transformer and Mamba style baselines on short-context language modeling.

Long-context sequence modeling is central to modern foundation models across language, vision, speech, time series, and genomics. Despite the architectural flexibility of the foundation-model paradigm, state-of-the-art systems are still overwhelmingly based on the Transformer and its self-attention mechanism.

A useful lens is to describe modern sequence mixers by how they route information from the past and how they maintain memory over time. In many modern architectures, routing decisions are input-dependent: the model uses the current token and its context to decide which parts of the visible history to consult. Under this view, self-attention implements an input-dependent direct-read mechanism: at each position, it computes a query-dependent pattern of relevance over the visible context and uses it to read out information from selected past positions. This framing highlights attention’s key strength, a selection mechanism over variable support length, but also a structural limitation: the retrieval is performed in a single pass, without an internal feedback loop that would repeatedly incorporate past readouts into an evolving state. Separately, standard implementations are also computationally expensive at long contexts due to quadratic time/memory scaling.

In parallel, structured recurrent sequence models, especially state-space models (SSMs), which realize long-range dynamics through a latent state and an explicit feedback path, have re-emerged as a compelling alternative for long-context modeling. SSMs can be interpreted as modern descendants of classical dynamical systems and admit linear (or near-linear) scaling in sequence length. However, for information-dense discrete data, a persistent challenge is that stable feedback dynamics often exhibit rapid attenuation of distant information (commonly exponential forgetting), which can hinder integrating multiple far-apart evidence snippets under heavy distractors. Selective SSMs (e.g., Mamba) can conditionally slow this attenuation by modulating the effective transition (e.g., ssm,≈ on selected steps, “freeze time”), but this mechanism is input-dependent and can fail when relevant and irrelevant positions induce similar local representations, leading to preserving or overwriting the wrong content.

These perspectives suggest complementary long-context failure modes. Stable feedback dynamics can suffer from exponential forgetting. Attention, while input-dependent, can suffer from dilution: when attention mass is spread across a large effective support of competing tokens (e.g., many near-tied logits), individual weights, and thus per-token contributions and sensitivities, decrease roughly inversely with that support (often behaving like O(1/S_eff(t)), and in the worst case like O(1/ℓ) when the effective support grows proportionally with context length). In practice, both effects can limit reliable long-range evidence integration.

We introduce Sessa, a decoder architecture that injects input-dependent attention into a feedback (recurrent) path, combining direct-read input-dependent routing with stateful aggregation through the feedback channel. Viewed through a temporal routing lens, for a fixed source token and target position (lag ℓ = t − source), a single self-attention layer routes influence via a single routing step (a direct edge source → t), while chain-structured state-space recurrences propagate along the unique length-ℓ temporal chain. Sessa introduces route diversity within a single layer: its attention-induced feedback operator aggregates contributions over multiple internal routing depths (and, in dense patterns, many temporal paths), which can help sustain long-range sensitivity when routing is diffuse. Concretely, while self-attention corresponds to an input-dependent direct-read system (in the values), Sessa realizes an input-dependent feedback system: it maintains a latent state over unbounded horizons, while the feedback dynamics remain input-dependent via attention-based routing inside the loop (potentially over variable-support patterns). Intuitively, Sessa retains the representational benefits of recurrence for long-range accumulation while leveraging attention as an input-dependent mechanism within the feedback pathway.

Related architectural ideas have introduced recurrence or feedback into sequence modeling. These approaches span a variety of feedback constructions and are typically presented in architecture-specific terms. Our contribution is complementary but mathematically different: we propose a routing-induced systems perspective that separates how context produces routing/mixing coefficients from how those coefficients are composed over time, and we use this lens to relate input-dependent routing directly to long-context sensitivity and memory-decay behavior.