Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information

Bias and fairness have become central concerns in deploying AI systems across domains like NLP, vision, and recommendation engines. Traditional bias mitigation techniques often rely on linear assumptions or require extensive labeled data, limiting their effectiveness in complex, real-world scenarios. This paper introduces SAL, a spectral method leveraging singular value decomposition (SVD) to identify and project out directions in neural representations that encode protected attributes such as gender or race. The core idea is to analyze the covariance between representations and protected attributes, then remove the components with the smallest singular values, which correspond to minimal covariance and thus minimal bias. The method is extended to nonlinear settings via kernel techniques (kSAL), enabling the capture of complex bias relationships. Extensive experiments demonstrate SAL’s superiority over existing methods like INLP, achieving significant bias reduction while maintaining task accuracy. In word embeddings, SAL reduces gender bias scores by about 30% without harming semantic similarity. In NLP tasks such as profession classification and sentiment analysis, SAL maintains over 85% accuracy while halving bias metrics like TPR gap. Importantly, SAL performs well even with limited labeled data, requiring only a small fraction of protected attribute annotations to achieve substantial bias mitigation. The kernel extension further enhances nonlinear bias removal, reducing bias by over 50% in complex scenarios. Overall, SAL offers a simple, efficient, and versatile tool for fair representation learning, with broad implications for privacy-preserving AI and equitable decision-making. Future work will explore deep integration, adaptive kernels, and large-scale scalability, aiming to make bias control an integral part of AI deployment pipelines.