Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads

Over recent years, dual-encoder architectures have become central in tasks such as cross-modal retrieval, contrastive learning, and operator approximation. Models like CLIP, DeepONet, and knowledge graph completion leverage inner product similarity of separately encoded inputs. Despite their widespread success, these methods lack a unified theoretical framework guiding the choice of interaction modes, normalization strategies, and their limitations. Spectral analysis, rooted in operator theory, has shown promise in understanding the intrinsic complexity of functions, but its integration into practical model design remains limited. This gap motivates the development of a systematic theory that connects spectral properties with approximation, identifiability, and sample efficiency, aiming to unify diverse applications under a common foundation.

The core challenge lies in quantifying the expressive power of multiplicative dual-encoders, especially in high-dimensional or flat-spectrum scenarios. Existing methods often rely on heuristic normalization, leading to ambiguous representations due to gauge symmetry. This symmetry causes the learned encoders to be only identifiable up to linear transformations, complicating interpretability and model comparison. Furthermore, the spectral tail decay rate critically influences the number of modes needed for accurate approximation, but practical guidelines for model selection and regularization are lacking. These issues hinder the deployment of dual-encoder architectures in complex real-world tasks where spectral properties vary widely.

The paper introduces the class of low interaction rank functions, characterized by their interaction spectrum. It establishes spectral truncation as the intrinsic approximation limit, with the tail decay rate governed by target smoothness. The key innovation is the use of whitening normalization as a gauge fixing method, which guarantees mode identifiability up to permutation and sign, resolving the interpretability issue in contrastive models. The framework also quantifies the sample complexity as a sum of encoder complexities, contrasting with traditional multiplicative bounds. Additionally, the analysis of flat-spectrum targets reveals a fundamental error floor, providing a spectral decay criterion for model selection. These contributions collectively unify diverse dual-encoder methods and offer practical tools for model design.

• �� Define the interaction operator and spectrum for target functions in L2 spaces, establishing the low interaction rank class.
• �� Decompose approximation error into spectral truncation and encoder realization errors, linking decay rates to target smoothness.
• �� Formalize gauge symmetry as linear transformations acting on encoders, and introduce normalization schemes.
• �� Prove whitening as a gauge fixing method that guarantees mode recovery up to permutation and sign, under spectral gap conditions.
• �� Analyze sample complexity via covariance estimation errors, deriving bounds dependent on spectral gap and sample size.
• �� Validate predictions through experiments on synthetic kernels, operator learning, and CLIP models, measuring spectral decay, mode recovery, and error floors.

Experiments involve synthetic kernel functions with known spectral decay, assessing approximation errors and mode recovery under different normalization schemes. Operator learning tasks test spectral decay and regularization effects on learned operators. CLIP models are analyzed for spectral similarity across independently trained pairs, with whitening applied to recover interpretable concept axes. The experiments vary target smoothness, spectral gap, and sample size to evaluate the theoretical bounds, confirming the spectral decay rates, the effectiveness of whitening in mode recovery, and the existence of a flat-spectrum error floor. Hyperparameters are tuned to match the theoretical regimes, and ablation studies compare normalization schemes' residual gauge groups and conditioning.

Spectral decay rates observed in experiments match the predicted polynomial or exponential decay, validating the spectral analysis framework. Whitening consistently recovers the true interaction modes, reducing gauge ambiguity and enhancing interpretability. In CLIP models, the interaction spectra are shared across different training runs, and the residual rotation after whitening aligns concept axes with human-interpretable semantics. Flat-spectrum targets exhibit an irreducible error floor proportional to 1−d/N, confirming the theoretical limitations. The sample complexity bounds derived from spectral gaps accurately predict the number of samples needed for reliable mode estimation, demonstrating the practical utility of the spectral decay criterion.

The framework guides the design of multimodal retrieval systems, knowledge graph completion, and operator learning models by informing the choice of interaction rank and normalization strategy. It enables practitioners to estimate the spectral properties of their target functions from data, optimizing model capacity and regularization accordingly. The spectral decay criterion helps determine when low-rank heads are sufficient or when early-interaction architectures are preferable, especially in high-dimensional or flat-spectrum scenarios. The insights also support interpretability efforts by ensuring mode recovery and reducing gauge ambiguity, fostering more transparent AI systems.

The analysis assumes target functions are smooth or analytic, which may not hold in real-world noisy or non-smooth data. Covariance estimation errors can impair mode recovery, especially with limited samples. Flat-spectrum targets impose an unavoidable error floor, limiting performance in certain applications. Extending the theory to non-linear or dynamic spectral scenarios, as well as handling non-stationary data, remains an open challenge. Future work should focus on robust spectral estimation and adaptive regularization to address these limitations.