Masked Video and Body-worn IMU Autoencoder for Egocentric Action Recognition
Introducing EVI-MAE, combining video and IMU data, achieving 92.78% accuracy in action recognition.
Key Findings
Methodology
The study proposes a multimodal Masked Autoencoder (MAE) method for action recognition by integrating body-worn IMU sensor data with egocentric video. It leverages collaborative dynamics among IMU devices, embedding relative motion features of human joints into a graph structure, and achieves strong multimodal representations through self-supervised pretraining.
Key Results
- Result 1: On the CMU-MMAC dataset, EVI-MAE method improved action recognition accuracy to 87.96%, significantly outperforming baseline models.
- Result 2: On the WEAR dataset, EVI-MAE achieved 92.78% accuracy, surpassing existing multimodal methods.
- Result 3: Ablation studies confirm the efficacy of constructing the IMU feature graph, even without pretraining.
Significance
This study significantly enhances action recognition accuracy through innovative multimodal fusion, especially in scenarios with partial IMU device loss or video quality degradation. It offers flexible usage in real-world applications and advances the field of multimodal action recognition.
Technical Contribution
Technical contributions include designing the first multimodal masked autoencoder combining egocentric video and body-worn IMU, proposing a graph-based IMU modeling technique, optimizing collaborative relationships among multiple IMU devices, and enhancing recognition robustness.
Novelty
This study is the first to embed IMU features into a graph structure and use multimodal masked autoencoder pretraining, significantly improving recognition accuracy and robustness, showing substantial innovation compared to existing methods.
Limitations
- Limitation 1: Recognition performance may decline in scenarios where IMU devices are completely missing.
- Limitation 2: Method performance may be limited under extremely poor video quality.
Future Work
Future work could explore more collaborative relationships among IMU devices, optimize graph structure construction, and validate method effectiveness in more real-world scenarios.
AI Executive Summary
This study introduces an innovative method for action recognition by integrating body-worn IMU sensor data with egocentric video through a multimodal masked autoencoder (EVI-MAE). Traditional action recognition methods often rely on single-modal data, struggling with complex environmental changes. EVI-MAE significantly improves recognition accuracy by embedding IMU features into a graph structure and leveraging collaborative dynamics among devices.
Experimental results show that the method performs exceptionally well on multiple public datasets, maintaining high recognition accuracy even in scenarios with partial IMU device loss or video quality degradation. This offers flexible usage in real-world applications and advances the field of multimodal action recognition.
Despite its excellent performance in many aspects, recognition performance may be limited in scenarios where IMU devices are completely missing or video quality is extremely poor. Future work could explore more collaborative relationships among IMU devices, optimize graph structure construction, and validate method effectiveness in more real-world scenarios.
Deep Analysis
Background
Action recognition is a crucial field for understanding human behavior, gaining focus with the development of wearable cameras and egocentric video datasets. However, recognizing actions solely from video poses challenges like viewpoint limitations and lighting variations. Recently, integrating multimodal signals has become key to addressing these issues, with IMU sensors gaining attention for their cost-effectiveness and efficiency.
Core Problem
Traditional action recognition methods rely on video data, struggling with lighting variations and viewpoint limitations. IMU sensors can capture accurate motion signals, but their potential remains underexplored. Effectively integrating video and IMU data to enhance recognition accuracy and robustness is a pressing challenge.
Innovation
This study innovatively proposes a multimodal masked autoencoder (EVI-MAE), enhancing action recognition performance through self-supervised pretraining by combining video and IMU data. Compared to existing methods, this approach first embeds IMU features into a graph structure, optimizing collaborative relationships among devices.
Methodology
- �� Design EVI-MAE for self-supervised pretraining to obtain multimodal representations.
- �� Embed IMU features into a graph structure to enhance collaborative relationships.
- �� Utilize masking strategies to enhance complementarity between video and IMU data.
- �� Employ multimodal encoders and decoders for feature reconstruction.
Experiments
Experiments use CMU-MMAC and WEAR datasets to evaluate method performance in various scenarios. Baseline model comparisons and ablation studies validate EVI-MAE's effectiveness. Key parameters include video and IMU masking ratios and graph structure construction methods.
Results
Results show EVI-MAE outperforms existing methods across multiple datasets, maintaining high recognition accuracy even with partial IMU device loss or video quality degradation. Ablation studies further confirm the efficacy of constructing the IMU feature graph.
Applications
The method is suitable for scenarios requiring high robustness in action recognition, such as sports monitoring and rehabilitation training. By combining video and IMU data, it maintains high recognition accuracy in complex environments.
Limitations & Outlook
Despite EVI-MAE's excellent performance in many aspects, recognition performance may be limited in scenarios where IMU devices are completely missing or video quality is extremely poor. Future work could explore more collaborative relationships among IMU devices and optimize graph structure construction.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, where video acts like your eyes and IMU sensors like your hands and feet. Your eyes can see ingredients, but sometimes lighting is poor or obstructed, while your hands and feet can sense chopping and walking motions. This study combines eye and hand-foot information, allowing accurate recognition even when lighting is poor or obstructed.
ELI14 Explained like you're 14
Hey, friends! Imagine you're playing a game, where your hand movements and screen visuals combine to help you defeat enemies faster! This study is like a game superpower, combining body motion sensors and video to make action recognition stronger! Even when video is unclear, sensors help accurately recognize actions!
Glossary
Masked Autoencoder (MAE)
A model using masking strategies for self-supervised learning, capable of learning effective representations from unlabeled data.
Used for pretraining by combining video and IMU data.
Inertial Measurement Unit (IMU)
A sensor capable of measuring acceleration, angular velocity, and orientation, commonly used for action recognition.
Used to capture human motion signals.
Egocentric Video
Video captured from the perspective of a camera-wearing person, commonly used for action recognition.
Combined with IMU data for action recognition.
Graph Structure
A data structure representing relationships between nodes and edges, commonly used for modeling complex relationships.
Used to model collaborative relationships among IMU devices.
Ablation Study
An experimental method evaluating the impact of removing or altering model components on overall performance.
Used to verify the impact of constructing the IMU feature graph on recognition performance.
Open Questions Unanswered questions from this research
- 1 How to maintain high recognition performance when IMU devices are completely missing? Current methods struggle with this issue, requiring exploration of new data fusion strategies.
- 2 How to optimize video and IMU data fusion in extreme environments? More robust model structures are needed.
Applications
Immediate Applications
Sports Monitoring
By combining video and IMU data, provide more accurate sports monitoring, suitable for athletes and rehabilitation patients.
Long-term Vision
Smart Home
Apply this technology to smart home systems for more precise action recognition and control, enhancing user experience.
Abstract
Compared with visual signals, Inertial Measurement Units (IMUs) placed on human limbs can capture accurate motion signals while being robust to lighting variation and occlusion. While these characteristics are intuitively valuable to help egocentric action recognition, the potential of IMUs remains under-explored. In this work, we present a novel method for action recognition that integrates motion data from body-worn IMUs with egocentric video. Due to the scarcity of labeled multimodal data, we design an MAE-based self-supervised pretraining method, obtaining strong multi-modal representations via modeling the natural correlation between visual and motion signals. To model the complex relation of multiple IMU devices placed across the body, we exploit the collaborative dynamics in multiple IMU devices and propose to embed the relative motion features of human joints into a graph structure. Experiments show our method can achieve state-of-the-art performance on multiple public datasets. The effectiveness of our MAE-based pretraining and graph-based IMU modeling are further validated by experiments in more challenging scenarios, including partially missing IMU devices and video quality corruption, promoting more flexible usages in the real world.