What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
Introduces ARGO1M dataset and CIR method for cross-scenario, cross-location action recognition, outperforming prior approaches.
Key Findings
Methodology
This paper presents ARGO1M, a large-scale dataset with 1.1 million video clips across 10 scenarios and 13 locations, designed to evaluate domain generalization in action recognition. The proposed Cross-Instance Reconstruction (CIR) method models each video as a weighted combination of other domain videos, leveraging cross-modal pairing with text narrations. The approach employs a dual loss framework: contrastive learning for video-text alignment and classification loss for action recognition. Video features are extracted using SlowFast, while text features utilize a frozen CLIP encoder. During training, the model learns to reconstruct each video from others via attention mechanisms, promoting domain-invariant representations. At inference, only the encoder and classifier are used, enabling recognition in unseen scenarios and locations without domain labels.
Key Results
- On ARGO1M’s 10 test splits, CIR consistently outperforms prior domain generalization methods, with an average accuracy boost of 2.1%, reaching up to 4.9% in challenging splits. The method shows robustness especially in splits with high domain shift, such as Knitting in India, where accuracy improvements are more pronounced. Ablation studies reveal that combining video-text pairing with cross-instance reconstruction yields the best performance, surpassing baselines like ERM, CORAL, DANN, and Mixup.
- CIR achieves these gains without relying on explicit domain labels, instead leveraging multimodal data. The results demonstrate the effectiveness of the reconstruction-based approach in capturing the core action semantics invariant to scene and location variations, significantly advancing the state-of-the-art in video domain generalization.
- Further experiments show that the method remains robust across different reconstruction strategies and hyperparameters, confirming its stability and adaptability. The comprehensive evaluation across diverse splits underscores its potential for real-world applications where environment variability is high.
Significance
This work addresses a critical challenge in deploying action recognition systems in real-world, unconstrained environments. The ARGO1M dataset fills a gap by providing a large-scale benchmark for scenario and location shifts, facilitating future research. The CIR method introduces a novel paradigm—cross-instance, multimodal reconstruction—to learn domain-invariant features, which could influence broader fields like multi-modal learning and domain adaptation. Its ability to generalize without domain labels reduces reliance on costly annotations, making it highly practical for applications like surveillance, robotics, and AR/VR. Overall, this research pushes the boundary of what’s possible in robust, adaptable video understanding, with implications for both academia and industry.
Technical Contribution
The core innovation lies in representing each video as a weighted combination of other domain videos through attention mechanisms, coupled with video-text pairing for semantic guidance. Unlike traditional distribution matching or adversarial domain adaptation, CIR emphasizes instance-level reconstruction, promoting semantic invariance. The integration of contrastive learning with reconstruction objectives enhances the model’s ability to capture action semantics across diverse environments. The ARGO1M dataset, with its extensive scenario and location diversity, provides a challenging benchmark that validates the method’s scalability and robustness. This combination of multimodal, instance-based reconstruction and large-scale data collection constitutes a significant technical advancement in domain generalization for video tasks.
Novelty
This is the first work to combine cross-instance attention-based reconstruction with multimodal video-text pairing specifically for domain generalization in action recognition. The ARGO1M dataset, with its unprecedented scale and diversity, enables rigorous evaluation of scenario and location shifts simultaneously. Unlike prior methods relying on explicit domain labels or distribution alignment, CIR learns to generalize by reconstructing videos from other domains, guided by textual descriptions, which provides richer semantic supervision. This approach introduces a new paradigm that leverages multimodal cues and instance-level reconstruction, setting a new direction for robust video understanding.
Limitations
- The approach depends heavily on high-quality textual descriptions; noisy or missing narrations could impair performance.
- Training complexity and computational costs are high due to multi-task objectives and large-scale data, limiting real-time deployment.
- The method’s effectiveness on other tasks like action prediction or long-term temporal modeling remains to be validated, and its robustness in extremely unseen environments needs further testing.
Future Work
Future research will explore integrating more sophisticated multimodal fusion techniques, such as transformer-based cross-modal attention, to further improve generalization. Expanding ARGO1M with more diverse scenarios and languages will test the method’s adaptability across cultures. Additionally, combining CIR with self-supervised learning and reinforcement learning could enable models to learn more autonomously from unlabelled data, broadening its applicability in real-world, dynamic environments.
AI Executive Summary
Action recognition models have achieved remarkable success in controlled settings, yet their performance often deteriorates dramatically when faced with unseen environments. This challenge stems from the inherent domain shift caused by variations in scene, tools, and background across different scenarios and locations. To address this, the authors introduce ARGO1M, a comprehensive dataset comprising 1.1 million video clips spanning 10 scenarios and 13 locations, designed explicitly to evaluate and improve domain generalization in action recognition. The dataset’s scale and diversity fill a critical gap, enabling rigorous benchmarking of models' robustness to environmental variability.
Building upon this foundation, the authors propose Cross-Instance Reconstruction (CIR), a novel method that models each video as a weighted combination of other domain videos. This reconstruction is guided by cross-modal pairing with textual narrations, which serve as semantic anchors. The approach employs attention mechanisms to compute similarity weights among videos, enabling the model to learn representations that capture the core action semantics invariant to scene and background changes. During training, the model optimizes a combination of contrastive losses for video-text alignment and classification losses for action recognition, fostering a rich, multimodal, domain-invariant feature space.
Experimental results on ARGO1M demonstrate that CIR outperforms existing domain generalization techniques such as ERM, CORAL, DANN, and Mixup across all test splits, with an average accuracy improvement of 2.1%. Notably, the method shows significant gains in challenging splits with high domain shift, validating its robustness. Ablation studies confirm that leveraging textual descriptions and multi-task learning are key to its success. These findings suggest that multimodal, instance-level reconstruction offers a promising pathway for achieving reliable action recognition in diverse, real-world environments.
This research has broad implications, advancing the field of video understanding and domain adaptation. By reducing dependence on domain labels and harnessing multimodal cues, CIR paves the way for more adaptable AI systems capable of functioning effectively across varied settings. Future work will focus on expanding dataset diversity, refining multimodal fusion strategies, and exploring applications in robotics, surveillance, and augmented reality, ultimately moving toward truly intelligent, context-aware visual systems.
Deep Dive
Plain Language Accessible to non-experts
想象你在不同国家的厨房里做饭，厨师们用不同的工具、不同的食材，背景也各不相同，但他们都在做“切菜”这个动作。传统的电脑就像只认识某个厨师用的刀和厨房环境，遇到不同国家的厨房就认不出来了。这个研究就像教电脑理解“切菜”的本质，不管用什么工具、在哪个厨房。科学家们用一种特别的方法，把每个视频都像用别的厨房的厨师帮忙“重建”一样，把不同厨房的场景拼在一起，让电脑学会只看动作本身。这样，不管是在印度还是意大利，电脑都能认出“切菜”。他们还用文字描述动作，帮助电脑更懂意思。最终，这个方法让电脑变得更聪明，能在各种不同环境中都认出动作，就像人一样灵活。
ELI14 Explained like you're 14
想象你在学校学跳舞，不同的老师教不同的动作，但你总能认出“跳舞”这个动作。现在，想让电脑也学会这个本领就很难，因为每次环境不同，动作可能看起来不一样。这个研究就像教电脑理解“跳舞”的本质，不管在哪个舞台、用什么音乐。科学家们用一种特别的办法，把每个跳舞的视频都像是用别的舞台帮忙“拼凑”一样，把不同的环境拼在一起，让电脑学会只看动作本身。这样，不管是在家里还是在学校，电脑都能认出“跳舞”。他们还用文字描述动作，帮助电脑更懂意思。最后，电脑变得更聪明，能在各种不同的场景中都认出动作，就像我们一样灵活。这就像你在不同的学校都能认出老师在教什么舞蹈一样，超级厉害！
Abstract
We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits. Code and data: https://chiaraplizz.github.io/what-can-a-cook/.