Unlocking Compositional Generalization in Continual Few-Shot Learning
Proposes COMPOSE framework: training preserves object geometry, inference dynamically composes slots, achieving state-of-the-art generalization in continual few-shot learning.
Key Findings
Methodology
The approach leverages patch-level semantic geometry from self-supervised ViT, employing a dual-phase strategy: during training, optimize holistic class identity to maintain object geometry; during inference, dynamically compose preserved slots for novel scenes. The training phase involves a lightweight router and projection head, optimizing class-level cross-entropy and cross-correlation regularization, while freezing the backbone to prevent drift. In inference, no gradient updates occur; scene matching uses bidirectional Chamfer distance on relative slot features. This design prevents overfitting to training patterns and enhances generalization to unseen concepts.
Key Results
- On CGQA and COBJ datasets, COMPOSE achieves 94.14% and 89.71% accuracy on unseen concepts, surpassing all baselines. It demonstrates superior zero-shot generalization and minimal catastrophic forgetting in continual learning benchmarks such as MiniImageNet, CIFAR100, and CUB200, with performance gains over 5%. Ablation studies confirm the importance of phase separation and geometric preservation. The method's robustness is validated across different backbone architectures, including DINOv2 ViT-B/14 and iBOT ViT-B/16.
- Compared to prior methods like RanPAC, EASE, and CoFiMA, COMPOSE maintains high accuracy on unseen concepts while exhibiting negligible forgetting, highlighting the effectiveness of decoupling representation learning from compositional inference. The results indicate that preserving object geometry during training and dynamic scene composition at inference are key to achieving strong generalization.
- Additional experiments show that the cross-correlation regularizer effectively prevents slot collapse, ensuring diverse and meaningful object representations. The approach also scales well across datasets with different scene complexities, confirming its broad applicability.
Significance
This work addresses fundamental limitations in current continual learning models, which often rely on global scene embeddings and struggle with unseen concept generalization. By decoupling representation learning from compositional inference, it introduces a paradigm shift that enhances the model's ability to recognize novel objects with minimal training data. The framework bridges the gap between object-centric vision and continual learning, offering a scalable solution for real-world applications like robotics, autonomous vehicles, and adaptive AI systems. Its robustness against catastrophic forgetting and strong zero-shot capabilities mark a significant advance in AI's lifelong learning capacity.
Technical Contribution
Key innovations include: 1) leveraging self-supervised ViT's patch-level geometric cues for object slot extraction; 2) phase separation—training focuses on holistic class identity, inference on compositional scene matching; 3) a cross-correlation regularizer to prevent slot collapse, ensuring diverse, reusable object representations. Unlike traditional end-to-end matching, this approach maintains geometric integrity and avoids overfitting to training patterns, providing theoretical guarantees for better generalization. The framework also introduces a novel attention-based readout for slot purity and a bidirectional Chamfer matching for scene composition.
Novelty
This is the first work to systematically integrate self-supervised ViT's patch-level geometry into a dual-phase continual learning framework, explicitly decoupling representation learning from compositional inference. It overcomes the limitations of existing slot attention methods that rely on recurrent states and global features, enabling robust generalization to unseen concepts. The phase separation strategy and geometric regularization constitute a new paradigm that significantly improves zero-shot and continual learning performance, setting a new state-of-the-art.
Limitations
- The approach heavily depends on the quality of self-supervised ViT features; if the backbone fails to produce coherent patch-level geometry, performance drops.
- Inference relies on non-gradient scene matching, which may struggle with highly complex or cluttered scenes requiring precise alignment.
- Training involves additional regularization and phase separation, increasing complexity and hyperparameter tuning requirements. Future work should focus on reducing computational overhead and enhancing robustness in diverse real-world scenarios.
Future Work
Future directions include integrating multi-modal cues (e.g., language, depth) to enrich object representations, developing adaptive mechanisms for scene complexity, and exploring meta-learning techniques to further improve few-shot generalization. Extending the framework to dynamic scenes and real-time applications, as well as reducing computational costs, are promising avenues for advancing lifelong learning systems.
AI Executive Summary
In the realm of visual recognition, models often struggle to learn new concepts without forgetting previous knowledge, especially in few-shot and continual learning settings. Traditional approaches tend to treat scenes as monolithic entities, which hampers their ability to generalize to novel objects, particularly when faced with unseen combinations. This limitation is rooted in the reliance on global scene embeddings and the overfitting of object representations to training patterns. To address this, the authors introduce COMPOSE, a novel framework that decouples representation learning from compositional inference. During training, the model leverages self-supervised Vision Transformers (ViTs) to extract patch-level semantic geometry, optimizing class identity with a lightweight router and projection head. This preserves object-level geometries, enabling the model to maintain highly generalizable features. In the inference phase, the preserved object slots are dynamically composed to match new scenes, using a bidirectional Chamfer distance on relative slot features. This phase separation prevents overfitting and representation drift, significantly enhancing the model’s ability to recognize unseen concepts. Extensive experiments on datasets like CGQA and COBJ demonstrate that COMPOSE achieves state-of-the-art results, with 94.14% and 89.71% accuracy on unseen concepts, respectively. It also exhibits minimal catastrophic forgetting across standard continual learning benchmarks such as MiniImageNet, CIFAR100, and CUB200, outperforming existing methods by a notable margin. The approach’s core strength lies in maintaining object geometry during training and employing a flexible, gradient-free scene composition during inference, which together foster robust generalization. This work marks a significant step forward in object-centric continual learning, opening pathways for more adaptable, lifelong AI systems capable of learning from limited data while retaining past knowledge. Future research will likely explore multi-modal integration, real-time scene understanding, and further reduction of computational complexity, pushing the boundaries of autonomous, lifelong learning agents.
Deep Dive
Abstract
Object-centric representations promise a key property for few-shot learning: Rather than treating a scene as a single unit, a model can decompose it into individual object-level parts that can be matched and compared across different concepts. In practice, this potential is rarely realized. Continual learners either collapse scenes into global embeddings, or train with part-level matching objectives that tie representations too closely to seen patterns, leaving them unable to generalize to truly novel concepts. In this paper, we identify this fundamental structural conflict and pioneer a new paradigm that strictly decouples representation learning from compositional inference. Leveraging the inherent patch-level semantic geometry of self-supervised Vision Transformers (ViTs), our framework employs a dual-phase strategy. During training, slot representations are optimized entirely toward holistic class identity, preserving highly generalizable, object-level geometries. At inference, preserved slots are dynamically composed to match novel scenes. We demonstrate that this paradigm offers dual structural benefits: The frozen backbone naturally prevents representation drift, while our lightweight, holistic optimization preserves the features' capacity for novel-concept transfer. Extensive experiments validate this approach, achieving state-of-the-art unseen-concept generalization and minimal forgetting across standard continual learning benchmarks.