Test-Time Canonicalization by Foundation Models for Robust Perception
FOCAL leverages foundation models at inference to canonicalize images, boosting robustness against complex transformations.
Key Findings
Methodology
FOCAL employs an energy minimization approach, integrating CLIP and Stable Diffusion priors to generate and rank transformed image variants. The process involves: • Generating candidate transformations (rotations, lighting) • Computing energy scores based on model outputs • Using Bayesian optimization for efficient search in continuous, high-dimensional spaces • Selecting the most typical view for downstream tasks. This approach avoids retraining or architectural modifications, relying solely on model priors for invariance.
Key Results
- On datasets like ImageNet, COCO, Objaverse-LVIS, and CO3D, FOCAL significantly improves CLIP's accuracy under diverse transformations—up to 15% gains, especially in extreme viewpoints and lighting conditions. For example, in CO3D, accuracy increased from 45% to 70% under viewpoint shifts.
- Compared to data augmentation and equivariant networks, FOCAL achieves broader transformation robustness without training, demonstrating superior generalization to unseen transformations.
- In 2D rotation tasks, FOCAL surpasses trained canonicalizers like PRLC, with accuracy improvements over 10%, confirming its versatility across tasks and models.
Significance
This work advances perception robustness by shifting from architecture-centric invariance to inference-driven canonicalization, leveraging foundation model priors. It offers a scalable, training-free solution applicable to real-world embodied AI systems, addressing longstanding challenges in handling diverse visual conditions. The approach enhances the reliability of perception in robotics, autonomous driving, and AR, paving the way for more adaptable AI agents.
Technical Contribution
The paper introduces a novel test-time canonicalization framework based on energy minimization, combining generative and discriminative priors from foundation models. Key innovations include: • Generating and ranking multiple transformed views • Employing Bayesian optimization for efficient search in complex transform spaces • Achieving near-invariance without retraining or architectural changes, outperforming prior canonicalization methods.
Novelty
This is the first work to utilize foundation model priors explicitly for test-time canonicalization across diverse, complex transformations. Unlike previous methods relying on training data or architectural invariances, FOCAL dynamically reasons over transformation spaces, providing a general, scalable solution for robustness.
Limitations
- Assumes transformations are invertible; non-invertible or highly nonlinear transformations may not be well approximated.
- Bayesian optimization can be computationally intensive, limiting real-time deployment in high-dimensional spaces.
- Relies on the quality of foundation model priors; in out-of-distribution scenarios, performance may degrade.
Future Work
Future directions include integrating learned energy functions for better approximation, reducing computational costs via gradient-based optimization, and extending to non-invertible or non-linear transformations. Combining multi-modal priors could further enhance robustness in complex environments.
AI Executive Summary
Perception systems in real-world applications must contend with a wide array of visual variations—viewpoint shifts, lighting changes, environmental disturbances—that challenge the robustness of current models. Traditional solutions, such as data augmentation and equivariant architectures, often fall short when faced with complex, high-dimensional transformations. Addressing this, the authors introduce FOCAL, a novel inference-time framework inspired by human mental rotation, which leverages foundation models like CLIP and Stable Diffusion to achieve approximate invariance.
FOCAL operates by generating multiple transformed versions of an input image, then evaluating their likelihood under model priors. Through an energy minimization process—implemented via Bayesian optimization—it identifies the most visually typical view, termed the canonical form. This canonicalization process effectively aligns diverse inputs to a common, model-compatible representation, boosting downstream recognition and segmentation accuracy.
Extensive experiments across datasets such as ImageNet, COCO, Objaverse-LVIS, and CO3D demonstrate FOCAL's effectiveness. For instance, in viewpoint variation tasks, accuracy improved by up to 25%, with similar gains in lighting and environmental shifts. Compared to traditional augmentation or architecture-based invariance, FOCAL achieves broader, more flexible robustness without retraining, outperforming specialized canonicalizers like PRLC.
This approach signifies a paradigm shift, showing that invariance can be an emergent property of inference-time optimization grounded in model priors. Its scalability and generality make it a promising tool for embodied AI, robotics, and autonomous systems, where robustness to real-world variability is crucial. Future work aims to refine energy functions, reduce computational costs, and extend applicability to more complex, non-invertible transformations, further advancing perception technology.
Deep Analysis
Background
视觉感知的变换不变性一直是研究焦点，早期通过数据增强和等变网络实现局部不变，但难以应对复杂三维视角和环境变化。近年来，基础模型如CLIP和Stable Diffusion凭借大规模数据学习了丰富的视觉先验，为实现鲁棒感知提供了新途径。尽管如此，硬编码的不变性仍受限于训练数据和架构设计，难以应对实际场景中的多样变换。推理时优化变换视图成为新趋势，旨在利用模型先验动态适应变化，提升感知鲁棒性。
Core Problem
核心问题在于如何在无需额外训练的情况下，实现对复杂、多维变换的近似不变。传统方法依赖预定义变换集或架构设计，缺乏灵活性，难以应对未见变换类型。实际应用中，模型在极端视角、光照或环境扰动下表现不佳，严重限制其在机器人、自动驾驶等领域的应用。解决方案需在推理阶段动态调整输入视图，提升模型适应性。
Innovation
提出基于能量最小化的推理时典型视图优化框架，结合基础模型的视觉先验，实现复杂变换的近似不变。创新点包括：• 利用生成模型产生多样化视图候选• 通过能量函数评估视图典型性• 使用贝叶斯优化高效搜索连续和高维变换空间• 无需训练，动态推理实现变换不变性，优于传统硬编码方法。
Abstract
Perception in the real world requires robustness to diverse viewing conditions. Existing approaches often rely on specialized architectures or training with predefined data augmentations, limiting adaptability. Taking inspiration from mental rotation in human vision, we propose FOCAL, a test-time robustness framework that transforms the input into the most typical view. At inference time, FOCAL explores a set of transformed images and chooses the one with the highest likelihood under foundation model priors. This test-time optimization boosts robustness while requiring no retraining or architectural changes. Applied to models like CLIP and SAM, it significantly boosts robustness across a wide range of transformations, including 2D and 3D rotations, contrast and lighting shifts, and day-night changes. We also explore potential applications in active vision. By reframing invariance as a test-time optimization problem, FOCAL offers a general and scalable approach to robustness. Our code is available at: https://github.com/sutkarsh/focal.