On Linear Identifiability of Learned Representations
Proposes that deep discriminative models are linearly identifiable in function space, validated on GPT-2, BERT, and simulated data, using nonlinear ICA theory.
Key Findings
Methodology
This paper constructs a broad family of discriminative models, formalized as softmax functions with deep neural network parameterizations. By imposing a diversity condition ensuring invertible difference matrices, the authors prove that, asymptotically, learned representations are identifiable up to a linear transformation in function space. The approach leverages recent nonlinear ICA results, analyzing the impact of model symmetries and parameterization. Empirical validation includes simulated classification tasks, CIFAR-10 self-supervised learning, and large-scale experiments with GPT-2 and BERT, measuring representation similarity via CCA and SVCCA. The methodology combines theoretical derivations with practical experiments to demonstrate the linear equivalence of learned representations across different models and training conditions.
Key Results
- In simulated K-way classification, two independently trained models' representations converged to a linear relationship, with CCA scores approaching 1 after sufficient training iterations, confirming the theoretical predictions. In real models, the last-layer representations of GPT-2 and BERT across different initializations showed high correlation (up to 0.98), especially as dataset size and model capacity increased. Self-supervised experiments on CIFAR-10 demonstrated a steady increase in linear similarity with more data and larger models, validating the asymptotic behavior. These results collectively support the claim that deep models' representations are asymptotically linearly identifiable in function space.
- The experiments reveal that increasing data volume, model size, and training duration enhances the linear similarity of learned representations, indicating that finite data and capacity models approximate the ideal asymptotic regime. The high correlation of final-layer representations across models suggests robustness and transferability. The findings imply that deep models, despite overparameterization, learn fundamentally similar features, facilitating reproducibility and interpretability. The validation across diverse tasks underscores the broad applicability of the theoretical framework.
Significance
This work fundamentally shifts the understanding of deep neural network representations by establishing their linear identifiability in function space, overcoming classical parameter space ambiguities. It provides a rigorous foundation for the reproducibility and transferability of learned features, crucial for practical deployment, model compression, and interpretability. The theoretical insights bridge the gap between nonlinear ICA and deep learning, offering a unifying perspective that explains empirical phenomena such as layer-wise representation similarity in large models like GPT-2 and BERT. The results pave the way for designing more reliable, interpretable, and robust models, especially in multi-task and multi-modal settings, where consistent representations are vital. Overall, this advances both theoretical understanding and practical methodologies in representation learning.
Technical Contribution
The paper introduces a formal framework for the linear identifiability of deep discriminative models in function space, grounded in nonlinear ICA theory. By defining a diversity condition on the data and model outputs, it proves that the learned representations are unique up to a linear transformation in the limit of infinite data and model convergence. This extends classical ICA results to complex neural network architectures, providing a theoretical guarantee for the stability and reproducibility of learned features. The approach also offers a practical method to measure representation similarity via CCA and SVCCA, validated through extensive experiments on simulated and real-world models, including GPT-2 and BERT. This work bridges a critical gap between theory and practice in deep representation learning.
Novelty
This research is the first to establish the asymptotic linear identifiability of deep neural network representations in function space, leveraging nonlinear ICA principles. Unlike prior work focusing solely on parameter space symmetries, it emphasizes the representation functions' uniqueness up to linear transformations, providing a more meaningful notion of equivalence. The integration of diversity conditions with neural network architectures and empirical validation on large-scale models like GPT-2 and BERT marks a significant advance. This fundamentally enhances understanding of why different models trained on similar data produce highly correlated features, addressing longstanding questions about the stability and interpretability of deep representations.
Limitations
- The theoretical guarantees rely on assumptions of infinite data and perfect model convergence, which are unattainable in practice. Finite data and optimization imperfections may lead to deviations from ideal linear equivalence. The diversity condition, while mild, may be violated in highly constrained or small datasets, limiting applicability. The computational cost of large-scale experiments and the reliance on linear correlation metrics like CCA/SVCCA may not capture all aspects of representational similarity. Extending the theory to account for finite-sample effects and non-ideal training remains an open challenge. Future work should focus on robustness under realistic conditions and practical algorithms for enforcing the conditions.
Future Work
Future research will explore finite-sample bounds and convergence rates, aiming to quantify how quickly representations approach the ideal linear equivalence. Extending the framework to unsupervised and semi-supervised settings, as well as multi-modal models, is a key direction. Developing scalable algorithms for measuring and enforcing linear similarity in large models will facilitate practical deployment. Additionally, integrating this theory with interpretability and robustness techniques could lead to more transparent AI systems. Investigating the implications for transfer learning, continual learning, and model compression will further enhance the utility of the theoretical insights. Ultimately, the goal is to establish a comprehensive understanding of representation stability in real-world, resource-constrained scenarios.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在学校里，每次考试都用不同的题目，但老师希望你用一种统一的方法来理解所有题目。这个方法就像学习一套秘密的技巧，能帮你在不同题目之间找到共同点。即使每次你用的工具不同，比如用笔、用电脑，最终你用的技巧让你做题的方式看起来很相似。这篇论文就像发现了这些技巧背后的秘密规则：只要你掌握了这些规则，不管用什么工具，做题的思路都可以用一套简单的线性方法描述。这让你的学习变得更可靠，也更容易分享和迁移到新题目上。
Abstract
Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.