Iterative Erasure Count Is Not an Affine-Invariant Concept Dimension

The quest to interpret neural network representations has historically centered on probing methods, which assess whether certain concepts can be linearly decoded from features. Early works like linear probing and INLP aimed to quantify the dimensionality of concept encoding, assuming that the number of directions removed or identified correlates with the intrinsic complexity of the concept. As models grew larger and more distributed, researchers recognized that simple linear measures might not fully capture the underlying geometry. Recent advances introduced tools like cross-covariance analysis, guard ranks, and spectral attribute removal, aiming to better understand the distributed nature of concepts. Despite these developments, a persistent challenge remains: whether operational metrics like iteration count or cumulative rank truly reflect an intrinsic property of the representation or are artifacts of the measurement procedure. The debate is compounded by the fact that neural representations are subject to various transformations—rotations, shear, reparameterizations—that preserve information but alter geometric measures. This paper situates itself within this evolving landscape, seeking to clarify the invariance properties of these metrics and their dependence on the measurement process.

The core problem addressed is whether iterative erasure counts, such as the number of steps needed to nullify a concept via linear probes, can be considered intrinsic, affine-invariant measures of concept dimensionality. Existing practices often interpret these counts as reflecting the true complexity or distributedness of concepts within neural representations. However, the authors argue that these operational metrics are susceptible to affine transformations, such as shear, which preserve the information content but alter the geometric and procedural measures. This raises fundamental questions about the validity of using iteration counts as semantic indicators, especially since different parameterizations or measurement procedures can produce different counts despite the underlying concept remaining unchanged. Resolving this ambiguity is crucial for developing robust interpretability tools that accurately reflect the intrinsic properties of neural codes.

The paper's primary innovation lies in establishing that iteration-based measures like cumulative edit rank are not affine-invariant and can vary discretely under invertible transformations, even when population-level quantities remain fixed. It introduces a formal affine-equivariance theorem for the entire measurement trajectory, demonstrating that while the full process can be transported consistently, individual counts are procedure-dependent. The authors construct explicit counterexamples using Gaussian models, showing that the same concept can appear to have different dimensions depending on the measurement procedure. They also extend the analysis to finite-sample calibration experiments, revealing how reparameterizations influence stopping rules and trajectories. This work shifts the paradigm from viewing erasure counts as intrinsic properties to understanding them as procedure-dependent, geometric measures, fundamentally altering how interpretability metrics should be interpreted.

• �� Construct a population Gaussian model with a known rank-one concept variable S and independent noise N, generating labels Y and features X via an invertible matrix A. • Derive theoretical results showing that generating dimension, sufficient linear dimension, and guarding rank are invariant under affine transformations, while cumulative erasure counts can change discretely. • Prove the affine-equivariance of the full transported-metric trajectory when the metric, probe, regularizer, and tie-breaking are transported consistently, using matrix algebra and properties of positive-definite forms. • Develop counterexamples with synthetic Gaussian data, applying shear transformations to demonstrate changes in iteration counts and cumulative ranks without altering population quantities. • Conduct finite-sample experiments with Adam/QR calibration, applying controlled reparameterizations (e.g., shear, identity) to frozen features, and observe the effects on stopping rules and trajectories across multiple runs. • Analyze the impact of reparameterizations on the Euclidean trajectories and stopping behavior, validating the theoretical predictions in practical settings.

The experimental setup involves synthetic Gaussian data with known concept and noise variables to test theoretical claims. The models are subjected to affine transformations, particularly shear, to observe changes in iteration counts and cumulative ranks. The experiments include fitting linear probes using Moore–Penrose least squares and full QR methods, measuring the stopping counts and ranks before and after transformations. Additionally, real neural features from models like V-JEPA2 and DINOv2 are used, with controlled reparameterizations applied to frozen features. The Adam/QR calibration involves multiple runs (20 per condition) to assess the variability in stopping behavior under different transformations. The experiments also examine the influence of different regularizers and tie-breaking rules, confirming the procedure dependence of these metrics. Results consistently show that while population quantities remain invariant, operational measures like erasure count and cumulative rank are highly sensitive to the measurement procedure and parameterization.

The experiments confirm that in Gaussian models, the generating dimension, sufficient linear dimension, and guarding rank are invariant under affine transformations, but the cumulative erasure count can change from 1 to 2 after shear. For the full QR procedure with two outputs, the cumulative edit rank shifts from 2 to 4, despite fixed model-defined quantities. In finite-sample Adam/QR calibration, identity mixing halts after a single update, whereas shear transformations accept multiple updates, demonstrating procedure dependence. Reparameterizations of frozen features in real neural models alter subsequent Euclidean trajectories without changing zero-order predictions, emphasizing the procedural influence on measurement paths. These results collectively establish that operational metrics are not intrinsic properties but depend on the measurement procedure and geometric transformations.

The findings have significant implications for neural interpretability, suggesting that current metrics like iteration counts and erasure ranks should be interpreted with caution. They highlight the necessity of developing geometry-aware, invariant measures that reflect true concept dimensions. In practical terms, researchers can improve interpretability by incorporating affine-equivariant metrics and considering the measurement procedure's influence. This approach can enhance robustness in model debugging, feature attribution, and understanding distributed representations. Long-term, these insights could lead to standardized protocols for measuring concept encoding, reducing the risk of overestimating distributedness or misinterpreting neural codes, thereby advancing the development of transparent and trustworthy AI systems.

The theoretical framework relies heavily on Gaussian assumptions and linearity, which may not fully capture the complexities of real neural networks with nonlinear activations and interactions. Experimental validation is primarily conducted on synthetic data and specific feature extractors, limiting generalizability. The procedure dependence of metrics poses challenges for practical measurement, especially in large-scale models where controlling all factors is difficult. The analysis focuses on affine transformations, leaving open questions about more general nonlinear reparameterizations. Future work should extend the theory to nonlinear settings, explore more robust metrics, and validate findings across diverse architectures and tasks.