Multimodal Knowledge Distillation for Egocentric Action Recognition Robust to Missing Modalities
KARMMA framework achieves robust egocentric action recognition under missing modalities, reducing computational resources by 50%.
Key Findings
Methodology
The KARMMA framework distills knowledge from a multimodal teacher model to a multimodal student model, which maintains high accuracy even with missing modalities. The teacher model fuses features from frozen unimodal encoders, while the student model uses smaller feature extractors and fusion blocks to reduce computational costs.
Key Results
- On the Epic-Kitchens dataset, the KARMMA student model achieves 43% accuracy when all modalities are available, a 3% improvement over the baseline.
- On the Something-Something dataset, the student model improves accuracy by 36.74% under missing modality conditions.
- In a 90% missing modality scenario, the KARMMA student model improves accuracy by 2% over the baseline on the Epic-Kitchens dataset.
Significance
This research significantly enhances the ability of robots to recognize actions under missing modality conditions, reducing reliance on a single modality and making it applicable for real-world applications with diverse sensor configurations, especially in dynamic or uncertain environments.
Technical Contribution
Introduces a multimodal-to-multimodal distillation framework that does not require modality alignment, combining modality dropout and a parameter-free token reduction strategy to significantly reduce computational costs and improve model flexibility.
Novelty
KARMMA is the first multimodal distillation framework that does not require modality alignment during training and inference, significantly enhancing robustness under missing modality conditions.
Limitations
- In some cases, audio and object detection annotations contribute less to action recognition, potentially leading to accuracy drops.
- The model's performance on a single modality is slightly lower than models optimized specifically for that modality.
Future Work
Future work could explore the impact of more modality combinations and test the framework's robustness in more complex sensor environments.
AI Executive Summary
Egocentric action recognition plays a crucial role in human-robot interaction, yet existing methods often rely on RGB videos, overlooking the potential of multimodal information. The KARMMA framework addresses performance degradation due to missing modalities through multimodal knowledge distillation.
The core of the KARMMA framework is distilling knowledge from a multimodal teacher model to a student model, which maintains high accuracy even with missing modalities. By fusing features from frozen unimodal encoders, KARMMA significantly reduces computational costs, making it suitable for deployment on robots.
Experimental results show that KARMMA performs excellently on both the Epic-Kitchens and Something-Something datasets, especially under missing modality conditions, significantly outperforming baseline models. This research provides new insights into multimodal action recognition with broad application potential.
Deep Analysis
Background
With the increase in human-robot interactions, egocentric action recognition has become an important research area. Traditional methods mainly rely on RGB videos, but multimodal information such as audio and optical flow can improve accuracy in complex environments. Recently, the release of multimodal datasets has driven the development of multimodal recognition methods.
Core Problem
Existing multimodal methods often assume all modalities are available during inference, but in real-world applications, sensor failures or privacy constraints often lead to missing modalities, affecting recognition accuracy.
Innovation
The KARMMA framework addresses the issue of missing modalities through multimodal-to-multimodal knowledge distillation. Innovations include a distillation process without modality alignment, a modality dropout strategy, and a parameter-free token reduction strategy.
Methodology
- �� Use frozen unimodal encoders to build the teacher model
- �� Distill knowledge to a lighter student model
- �� Student model uses smaller feature extractors and fusion blocks
- �� Apply modality dropout strategy to enhance robustness
- �� Employ parameter-free token reduction strategy to reduce computational costs
Experiments
Experiments were conducted on the Epic-Kitchens and Something-Something datasets, using RGB video, optical flow, and audio as modalities. Baseline models were compared with the KARMMA student model to evaluate performance under different modality combinations.
Results
The KARMMA student model achieves 43% accuracy on the Epic-Kitchens dataset, a 3% improvement over the baseline. On the Something-Something dataset, accuracy improves by 36.74% under missing modality conditions.
Applications
KARMMA is suitable for robotic applications requiring multi-sensor configurations, especially in dynamic or uncertain environments, maintaining efficient action recognition under missing modality conditions.
Limitations & Outlook
While KARMMA performs well under missing modality conditions, its performance on a single modality is slightly lower than models optimized specifically for that modality. Additionally, audio and object detection annotations contribute less in some scenarios, potentially affecting accuracy.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, usually using your eyes to see ingredients and ears to hear the sound of boiling water. KARMMA is like a smart kitchen assistant that can help you determine if the food is cooked even if you close your eyes, by using hearing and touch. This method allows robots to function normally even when some sensors fail.
ELI14 Explained like you're 14
Imagine you're playing a game, usually using your eyes to see the screen and ears to hear the sound. KARMMA is like a super assistant that can help you keep playing even if you close your eyes, by using sound and touch. This method allows robots to work normally even when some sensors fail.
Glossary
Knowledge Distillation
A method of transferring knowledge from a large model to a smaller one, often used to improve the performance of the smaller model.
Used in KARMMA to transfer knowledge from the multimodal teacher model to the student model.
Modality Dropout
A method of randomly dropping certain modalities during training to improve model robustness under missing modality conditions.
Used to train the KARMMA student model to maintain high performance even when modalities are missing.
Token Reduction
A method of reducing the number of input tokens to lower computational costs.
Used in KARMMA to reduce the computational load of the fusion block.
Egocentric Action Recognition
The task of recognizing actions from a first-person perspective, often used in human-robot interaction.
The main application scenario for the KARMMA framework.
Multimodal
Techniques involving multiple data modalities, such as visual and audio.
KARMMA framework leverages multimodal information to improve action recognition accuracy.
Open Questions Unanswered questions from this research
- 1 How can model robustness be further improved with more modality combinations?
- 2 How does KARMMA perform in more complex sensor environments?
Applications
Immediate Applications
Robotic Assistants
KARMMA can be used in household or industrial robots to help them accurately recognize and execute tasks even with missing modalities.
Long-term Vision
Intelligent Surveillance Systems
By combining multimodal information, KARMMA can be used to develop smarter surveillance systems that function even when sensors fail.
Abstract
Egocentric action recognition enables robots to facilitate human-robot interactions and monitor task progress. Existing methods often rely solely on RGB videos, although additional modalities, such as audio, can improve accuracy under challenging conditions. However, most multimodal approaches assume that all modalities are available at inference time, leading to significant accuracy drops, or even failure, when inputs are missing. To address this limitation, we introduce KARMMA, a multimodal Knowledge distillation framework for egocentric Action Recognition robust to Missing ModAlities that does not require modality alignment across all samples during training or inference. KARMMA distills knowledge from a multimodal teacher into a multimodal student that leverages all available modalities while remaining robust to missing ones, enabling deployment across diverse sensor configurations without retraining. Our student uses approximately 50% fewer computational resources than the teacher, resulting in a lightweight and fast model that is well suited for on-robot deployment. Experiments on Epic-Kitchens and Something-Something demonstrate that our student achieves competitive accuracy while significantly reducing performance degradation under missing modality conditions.