MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
Proposes MMG-Ego4D, a multimodal egocentric action recognition dataset and framework, with Transformer fusion, contrastive alignment, and prototypical loss, improving generalization in missing and zero-shot scenarios.
Key Findings
Methodology
This work introduces a Transformer-based fusion module combined with modality Dropout, contrastive feature alignment, and a cross-modal prototypical loss to enhance model robustness across missing and unseen modalities. The dataset MMG-Ego4D integrates video, audio, and IMU data, with rigorous annotation and split for few-shot evaluation. The training pipeline includes unimodal pretraining, multimodal fusion with contrastive alignment, and meta-learning with prototypical loss, supporting both supervised and zero-shot settings. Experiments demonstrate significant improvements over baselines, with accuracy reaching 58.89% in few-shot, and 12% gains in cross-modal zero-shot tasks.
Key Results
- On MMG-Ego4D, the proposed method outperforms baselines in missing modality and zero-shot tasks, achieving 58.89% accuracy in few-shot scenarios, with a 12% boost over previous methods. Ablation studies show that the fusion module, contrastive alignment, and prototypical loss contribute substantially to performance gains.
- In full-modal settings, the model achieves 78.3% average accuracy, surpassing state-of-the-art. The cross-modal prototypical loss notably improves recognition of unseen modalities, validating its effectiveness.
- Robustness tests confirm that modality Dropout and feature alignment significantly enhance model stability under partial modality conditions, indicating strong generalization capacity across diverse scenarios.
Significance
This research addresses critical gaps in egocentric action recognition, especially under real-world constraints where modalities may be missing or unseen. By establishing a comprehensive benchmark and dataset, it paves the way for more resilient multimodal perception systems, applicable in security, assistive technology, and privacy-sensitive environments. The innovations in fusion, alignment, and few-shot learning extend the theoretical understanding of multimodal generalization, offering practical solutions for deploying intelligent systems in complex, unpredictable settings.
Technical Contribution
The paper introduces a Transformer-based fusion architecture that supports flexible input modalities, combined with modality Dropout for robustness. It proposes a contrastive alignment loss to unify feature spaces across modalities, and a novel cross-modal prototypical loss for few-shot and zero-shot learning. These components collectively improve model adaptability and accuracy, representing a significant advancement over existing multimodal fusion and generalization techniques.
Novelty
This is the first comprehensive study to formalize the multimodal generalization (MMG) problem in egocentric action recognition, defining specific tasks for missing and unseen modalities. The creation of MMG-Ego4D dataset and the integration of Transformer fusion, contrastive alignment, and cross-modal prototypical loss constitute key innovations that distinguish this work from prior research focused mainly on full-modal performance or simple fusion strategies.
Limitations
- The model still struggles under extreme modality absence, such as only one modality available, indicating room for further robustness improvements.
- The dataset, while extensive, is limited to specific egocentric scenarios, raising questions about scalability to diverse real-world environments.
- Training complexity and computational costs are high, especially with multi-component loss functions, which may hinder deployment on resource-constrained devices.
Future Work
Future directions include scaling to larger, more diverse datasets, integrating self-supervised learning to reduce annotation dependency, and optimizing models for real-time deployment. Exploring adaptive fusion strategies and domain adaptation techniques will further enhance robustness in real-world applications.
AI Executive Summary
Egocentric action recognition has gained importance with the proliferation of head-mounted devices, yet existing models often rely on complete multimodal data, limiting their real-world applicability. In practical scenarios, modalities such as video, audio, or inertial sensors can be intermittently unavailable due to privacy, security, or technical constraints. Addressing this gap, the authors propose MMG-Ego4D, a novel benchmark and dataset designed to evaluate and improve the generalization ability of multimodal action recognition models under conditions of missing and unseen modalities.
The core innovation lies in a Transformer-based fusion framework that integrates features from video, audio, and IMU sensors. To enhance robustness, the model employs modality Dropout during training, randomly omitting modalities to simulate real-world conditions. Additionally, a contrastive alignment mechanism maps features from different modalities into a common space, facilitating knowledge transfer and feature sharing. For few-shot and zero-shot learning, a cross-modal prototypical loss guides the model to recognize new categories and modalities with limited data.
Experimental results on the newly constructed MMG-Ego4D dataset demonstrate that the proposed approach significantly outperforms baseline models, achieving 58.89% accuracy in few-shot scenarios and a 12% improvement in cross-modal zero-shot tasks. Ablation studies confirm that each component—fusion, alignment, and prototypical loss—contributes to the overall performance, validating the effectiveness of the design.
This work advances the field by providing a systematic framework for multimodal generalization, addressing practical challenges in egocentric perception. It offers a foundation for developing more resilient, privacy-aware, and adaptable intelligent systems, with broad implications for security, assistive technology, and human-computer interaction. Future research will focus on scaling to larger datasets, reducing computational costs, and deploying models in real-time environments, pushing the boundaries of multimodal perception in complex settings.
Deep Analysis
Background
多模态学习在动作识别领域逐步成为主流，尤其是在egocentric场景中，结合视频、音频和IMU数据能显著提升识别性能。早期研究多集中在单模态或双模态融合，如VGG、C3D、Transformer等架构，但在实际应用中，模态缺失和新模态识别仍是难点。近年来，跨模态对齐和少样本学习成为研究热点，推动模型鲁棒性提升。然而，现有数据集如Ego4D虽丰富，但未充分考虑模态缺失和零样本问题，限制了模型的实际应用潜力。本文基于此背景，提出专门针对模态缺失和未见模态的评估任务和数据集，旨在推动多模态感知的实际应用。
Core Problem
在实际场景中，egocentric动作识别面临模态不完整、未见模态和少样本等挑战。传统模型高度依赖全部模态信息，缺乏在模态缺失或新模态出现时的适应能力，严重影响鲁棒性。这不仅限制了模型的实用性，也阻碍了其在隐私保护、边缘计算等场景中的部署。如何设计能在模态缺失和零样本场景下保持高性能的模型，成为亟待解决的核心问题。该问题关系到模型的泛化能力和实际应用的可靠性。
Innovation
提出多模态泛化（MMG）问题，定义缺失模态和跨模态零样本两个新任务。引入Transformer融合模块，结合模态Dropout策略，增强模型对模态变化的鲁棒性。设计对比对齐机制，将不同模态的特征映射到统一空间，提升信息互补。创新性地提出跨模态原型损失，支持少样本和零样本学习，显著改善模型在新类别和新模态上的表现。这些技术突破为多模态学习提供新思路，推动模型在复杂环境中的适应性。
Methodology
- �� 单模态预训练：分别训练视频（MViT-B，基于Kinetics-400）、音频（AST，基于AudioSet）和IMU特征提取器。• 多模态融合：采用Transformer结构，融合不同模态特征，加入模态特定嵌入，随机Dropout增强鲁棒性。• 模态对齐：利用对比损失（NCE）将不同模态特征映射到共同空间，确保同一数据点的不同模态特征靠近。• 跨模态原型：构建类别原型，支持不同模态的支持和查询样本，通过距离实现分类。• 训练流程：包括无监督对齐、监督分类和元学习，支持少样本和零样本场景。
Experiments
在MMG-Ego4D上，设计全模态、缺失模态和零样本多场景评估。对比多种基线，验证融合、对齐和原型机制贡献。指标为Top-1准确率，少样本条件下视频识别达58.89%，验证鲁棒性。消融实验显示，Dropout和对齐提升模型稳定性。模型在不同模态组合下表现一致，验证泛化能力。
Results
方法在缺失模态和零样本任务中优于基线，少样本识别达58.89%，跨模态提升12%。融合和对齐贡献最大，验证技术有效。全模态准确率达78.3%，优于SOTA。未见模态识别能力提升12%，验证泛化潜力。
Applications
该技术适用于安全监控、智能助手和隐私保护场景，支持模态受限环境下的鲁棒识别。未来结合边缘计算，提升实时性和安全性，推动多模态系统实际部署。
Limitations & Outlook
在极端模态缺失（如只剩单一模态）时性能仍下降，训练成本较高。数据集偏向特定场景，泛化到更复杂环境仍需验证。未来需优化模型结构，降低计算复杂度，增强多源、多任务适应性。
Plain Language Accessible to non-experts
想象你在厨房里做饭，手里拿刀、勺子和锅，但有时候你会忘记带某样工具。即使缺少某个工具，你还是可以用其他工具完成任务，比如用勺子搅拌。这个研究就像教厨师如何在工具不全时，仍然能做出美味菜肴。它让电脑学会在不同的感官信息缺失时，依然能正确理解动作，比如只用声音或运动传感器，也能判断出你在切菜或炒菜。通过训练，电脑学会了像厨师一样灵活应变，不怕工具不全。这样，无论环境多复杂，系统都能保持准确，像个聪明的厨师一样，随时准备应对各种突发情况。
ELI14 Explained like you're 14
想象你在玩一个游戏，有时候你只看到画面，有时候只听到声音，甚至只感觉到手上的摇杆在动。即使你没有全部信息，你还是能猜出你在做什么，比如在跳舞、跑步或打架。这篇论文就像教电脑如何在只看到、只听到或只感觉到一部分信息时，依然能知道发生了什么。它用一种聪明的方法，把不同的感觉合在一起，让电脑变得更聪明、更灵活。比如，电脑可以用视频、声音和运动传感器的数据，学会识别动作。即使某个数据缺失，它也能用剩下的部分猜出正确的动作。这就像你在黑暗中用手摸东西，还是能知道那是什么。这样，未来的智能设备就能更好地理解我们，即使信息不完整，也能帮我们做事。
Glossary
Transformer（变换器）
一种基于注意力机制的深度学习模型，能有效融合多模态信息。
用于多模态融合模块，提升信息整合能力。
对比对齐（Contrastive Alignment）
通过对比损失将不同模态的特征映射到相似空间，增强模态间的关联。
实现模态特征的跨模态对齐。
跨模态原型（Cross-Modal Prototypical）
利用类别原型进行少样本和零样本分类，支持不同模态间的特征匹配。
支持新类别和新模态的识别。
模态Dropout（Modal Dropout）
随机在训练中丢弃部分模态，增强模型对模态缺失的鲁棒性。
在训练中应用，提升模型在实际场景中的适应能力。
MMG-Ego4D（多模态泛化-Ego4D）
一个针对第一人称动作识别的多模态数据集和评估任务，支持模态缺失和零样本测试。
作为本文的核心数据平台。
Open Questions Unanswered questions from this research
- 1 如何在极端模态缺失（如只剩单一模态）时保持性能？
- 2 模型在多源、多任务环境中的泛化能力如何进一步提升？
- 3 如何降低训练成本，适应边缘设备部署？
Applications
Immediate Applications
智能监控系统
在安全监控中，系统能在模态受限时依然准确识别异常行为，保障安全。
隐私保护设备
在隐私敏感场景下，模型可在不依赖全部模态的情况下进行动作识别，保护用户隐私。
Long-term Vision
多模态智能助手
未来的助手能在模态不全时依然理解用户意图，实现更自然交互。
Abstract
In this paper, we study a novel problem in egocentric action recognition, which we term as "Multimodal Generalization" (MMG). MMG aims to study how systems can generalize when data from certain modalities is limited or even completely missing. We thoroughly investigate MMG in the context of standard supervised action recognition and the more challenging few-shot setting for learning new action categories. MMG consists of two novel scenarios, designed to support security, and efficiency considerations in real-world applications: (1) missing modality generalization where some modalities that were present during the train time are missing during the inference time, and (2) cross-modal zero-shot generalization, where the modalities present during the inference time and the training time are disjoint. To enable this investigation, we construct a new dataset MMG-Ego4D containing data points with video, audio, and inertial motion sensor (IMU) modalities. Our dataset is derived from Ego4D dataset, but processed and thoroughly re-annotated by human experts to facilitate research in the MMG problem. We evaluate a diverse array of models on MMG-Ego4D and propose new methods with improved generalization ability. In particular, we introduce a new fusion module with modality dropout training, contrastive-based alignment training, and a novel cross-modal prototypical loss for better few-shot performance. We hope this study will serve as a benchmark and guide future research in multimodal generalization problems. The benchmark and code will be available at https://github.com/facebookresearch/MMG_Ego4D.
References (20)
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov et al.
Multiscale Vision Transformers
Haoqi Fan, Bo Xiong, K. Mangalam et al.
AST: Audio Spectrogram Transformer
Yuan Gong, Yu-An Chung, James R. Glass
Ego4D: Around the World in 3,000 Hours of Egocentric Video
K. Grauman, Andrew Westbury, Eugene Byrne et al.
The Kinetics Human Action Video Dataset
W. Kay, João Carreira, K. Simonyan et al.
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions
Chunhui Gu, Chen Sun, David A. Ross et al.
Comparing deep neural networks against humans: object recognition when the signal gets weaker
Robert Geirhos, David H. J. Janssen, H. Schütt et al.
Spatiotemporal Multiplier Networks for Video Action Recognition
Christoph Feichtenhofer, A. Pinz, Richard P. Wildes
A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions
Samuel F. Dodge, Lina Karam
Missing Modalities Imputation via Cascaded Residual Autoencoder
Luan Tran, Xiaoming Liu, Jiayu Zhou et al.
Appearance-and-Relation Networks for Video Classification
Limin Wang, Wei Li, Wen Li et al.
Ground-Truth Adversarial Examples
Nicholas Carlini, Guy Katz, Clark W. Barrett et al.
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Zhun Liu, Ying Shen, V. Lakshminarasimhan et al.
Self-Paced Adversarial Training for Multimodal Few-Shot Learning
Frederik Pahde, Oleksiy Ostapenko, P. Jähnichen et al.
Multimodal One-shot Learning of Speech and Images
Ryan Eloff, H. Engelbrecht, H. Kamper
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video
Yin Li, Miao Liu, James M. Rehg
Learning Factorized Multimodal Representations
Yao-Hung Hubert Tsai, P. Liang, Amir Zadeh et al.
Cross-modal Hallucination for Few-shot Fine-grained Recognition
Frederik Pahde, P. Jähnichen, T. Klein et al.
On First-Order Meta-Learning Algorithms
Alex Nichol, Joshua Achiam, John Schulman
Prototypical Networks for Few-shot Learning
Jake Snell, Kevin Swersky, R. Zemel
Cited By (20)
Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey
Exploring Missing Modality in Multimodal Egocentric Datasets
Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities
Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition
COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition
Towards Achieving Perfect Multimodal Alignment
RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generation
Cognition Transferring and Decoupling for Text-Supervised Egocentric Semantic Segmentation
Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition
Deep Multimodal Learning with Missing Modality: A Survey
EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs
C3T: Cross-modal Transfer Through Time for Sensor-based Human Activity Recognition
Empirical study of 3D-HPE on HOI4D egocentric vision dataset based on deep learning
Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition
EMAG: Ego-motion Aware and Generalizable 2D Hand Forecasting from Egocentric Videos
Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration
Relative Norm Alignment for Tackling Domain Shift in Deep Multi-modal Classification
Continual Multimodal Egocentric Activity Recognition via Modality-Aware Novel Detection
MMW-AQA: Multimodal In-the-Wild Dataset for Action Quality Assessment