LS-HAR: Language Supervised Human Action Recognition with Salient Fusion, Construction Sites as a Use-Case
LS-HAR leverages language-guided skeleton and visual features with Salient Fusion, achieving 89.5% accuracy on NTU-RGB+D XSub dataset.
Key Findings
Methodology
The LS-HAR framework integrates skeleton and visual modalities, guided by a pre-trained language model. It employs learnable prompts conditioned on skeleton features to enhance semantic encoding. The Salient Fusion module, based on attention and transformer mechanisms, dynamically prioritizes key frames and joints, effectively reducing high-dimensional feature redundancy. The model is trained end-to-end on multiple datasets, including a newly introduced construction-specific VolvoConstAct dataset, demonstrating robustness and superior performance in complex environments.
Key Results
- On NTU-RGB+D XSub, the model achieves 89.5% accuracy, surpassing previous SOTA by 1%. On NTU-RGB+D 120, it reaches 82.3%. In real-world construction scenarios with VolvoConstAct, the recognition accuracy is 85%, confirming robustness.
- Salient Fusion significantly improves multi-modal feature utilization by emphasizing critical frames and joints. Language-guided skeleton features enhance semantic understanding. Cross-dataset evaluations show strong generalization, outperforming existing methods.
- Ablation studies reveal that learnable prompts and Salient Fusion contribute substantially to performance gains, with prompts guiding skeleton encoding and attention mechanisms refining feature importance.
Significance
This work advances multi-modal HAR, especially in challenging construction environments. By integrating language supervision and attention-based fusion, it addresses key issues like environmental noise, occlusion, and complex actions. The approach enhances safety and efficiency in industrial automation, enabling robots to interpret human commands accurately. The introduction of a construction-specific dataset fills a critical gap, facilitating future research and industry deployment. Overall, it paves the way for more intelligent, context-aware robotic systems in real-world scenarios.
Technical Contribution
The main technical innovations include: 1) leveraging pre-trained language models with learnable prompts to supervise skeleton feature extraction; 2) designing a Salient Fusion module with cross-attention and transformer layers for dynamic feature prioritization; 3) constructing VolvoConstAct, a dataset tailored for construction site HAR, with multi-modal data including RGB, depth, and skeletons; 4) demonstrating end-to-end training with contrastive and classification losses, improving multi-modal integration and semantic understanding. These contributions significantly differ from prior pose or video-only methods.
Novelty
This is the first work to incorporate language models as global semantic guides for skeleton-based HAR in construction scenarios. The use of learnable prompts conditioned on skeleton features and the Salient Fusion mechanism for dynamic, attention-driven multi-modal fusion are novel. Additionally, creating a dedicated construction site dataset addresses industry-specific challenges, enabling more accurate and robust action recognition in real-world environments.
Limitations
- The model's performance degrades under severe occlusion or environmental disturbances, such as poor lighting or sensor noise, which limit real-world robustness.
- Generalization to multi-action or highly dynamic scenes remains challenging, requiring larger and more diverse datasets.
- Computational complexity and inference speed need optimization for deployment in real-time systems, especially on resource-constrained robots.
Future Work
Future directions include integrating additional sensors like depth and audio, exploring semi-supervised learning to reduce data annotation costs, and deploying the system on actual construction robots for real-time testing. Further research will focus on improving robustness under adverse conditions and reducing computational overhead to facilitate practical industrial applications.
AI Executive Summary
Human Action Recognition (HAR) is vital for autonomous systems, yet existing methods struggle in complex environments like construction sites due to environmental noise, occlusion, and diverse actions. Traditional approaches relying solely on skeleton or visual data often face limitations in accuracy and robustness. Recent advances in deep learning, especially transformer architectures, have enabled more effective multi-modal fusion, but challenges remain in efficiently integrating high-dimensional features and understanding semantic context.
This study introduces LS-HAR, a novel framework that leverages pre-trained language models to guide skeleton feature extraction via learnable prompts, enhancing semantic richness. The Salient Fusion module, based on attention and transformer mechanisms, dynamically emphasizes critical frames and joints, effectively reducing feature redundancy and improving multi-modal integration. A key innovation is the construction of VolvoConstAct, a dedicated dataset for construction site HAR, capturing RGB, depth, and skeleton data in real-world scenarios.
Experimental results across multiple datasets—including NTU-RGB+D, NTU-RGB+D 120, NW-UCLA, and VolvoConstAct—demonstrate the effectiveness of LS-HAR. On NTU-RGB+D XSub, the model achieves 89.5% accuracy, outperforming previous state-of-the-art methods by 1%. In construction scenarios, recognition accuracy reaches 85%, validating its robustness in practical applications. Ablation studies confirm the contributions of learnable prompts and Salient Fusion, highlighting their importance in performance gains.
The approach significantly advances multi-modal HAR, addressing key industry needs for safety and efficiency. By enabling robots to interpret human commands accurately in noisy, occluded environments, it opens pathways for fully autonomous construction machinery and industrial automation. Future work will focus on integrating additional sensors, optimizing inference speed, and deploying on real robots, pushing the boundaries of intelligent human-robot collaboration.
Deep Dive
Abstract
Detecting human actions is a crucial task for autonomous robots and vehicles, often requiring the integration of various data modalities for improved accuracy. In this study, we introduce a novel approach to Human Action Recognition (HAR) using language supervision named LS-HAR based on skeleton and visual cues. Our method leverages a language model to guide the feature extraction process in the skeleton encoder. Specifically, we employ learnable prompts for the language model conditioned on the skeleton modality to optimize feature representation. Furthermore, we propose a fusion mechanism that combines dual-modality features using a salient fusion module, incorporating attention and transformer mechanisms to address the modalities' high dimensionality. This fusion process prioritizes informative video frames and body joints, enhancing the recognition accuracy of human actions. Additionally, we introduce a new dataset tailored for real-world robotic applications in construction sites, featuring visual, skeleton, and depth data modalities, named VolvoConstAct. This dataset serves to facilitate the training and evaluation of machine learning models to instruct autonomous construction machines for performing necessary tasks in real-world construction sites. To evaluate our approach, we conduct experiments on our dataset as well as three widely used public datasets: NTU-RGB+D, NTU-RGB+D 120, and NW-UCLA. Results reveal that our proposed method achieves promising performance across all datasets, demonstrating its robustness and potential for various applications. The code, dataset, and demonstration of real-machine experiments are available at: https://mmahdavian.github.io/ls_har/