Just Add $π$! Pose Induced Video Transformers for Understanding Activities of Daily Living
PI-ViT integrates 2D/3D skeleton info during training to enhance video transformer-based activity recognition, achieving SOTA without extra inference cost.
Key Findings
Methodology
PI-ViT employs two plug-in modules—2D Skeleton Induction Module (2D-SIM) and 3D Skeleton Induction Module (3D-SIM)—to incorporate pose information during training. 2D-SIM maps skeleton joints to visual tokens, refining appearance features for fine-grained discrimination. 3D-SIM aligns features with viewpoint-invariant skeleton motion representations, enhancing motion understanding. Both modules perform auxiliary tasks—joint classification and feature alignment—during training, then are discarded at inference, incurring no additional computational overhead. The backbone is based on TimeSformer, trained on datasets like Toyota-Smarthome, NTU120, and NTU60, demonstrating superior performance over existing methods.
Key Results
- On Toyota-Smarthome, PI-ViT achieves the highest accuracy with only RGB input, improving cross-view accuracy by 6.9%, and surpasses prior SOTA on NTU120 and NTU60 by 1.4% and 1.1%, respectively.
- Introducing pose information notably improves recognition of subtle, fine-grained actions, especially in complex multi-view scenarios.
- During inference, the model maintains high accuracy without pose input, confirming the effectiveness of training-time pose guidance.
Significance
This work advances the field of video-based activity recognition by eliminating the need for pose data during inference, enabling more practical deployment in real-world scenarios like surveillance and assistive robotics. It addresses the challenge of recognizing similar actions from multiple viewpoints with high precision, crucial for applications requiring robust understanding of daily activities. The approach also opens avenues for efficient multi-modal training strategies that do not increase inference complexity, fostering scalable and cost-effective solutions.
Technical Contribution
The paper introduces a novel training paradigm where pose information guides feature learning via auxiliary tasks, enabling the model to internalize pose-invariant and fine-grained discriminative features. The design of plug-in modules allows flexible integration into existing video transformers, and the auxiliary tasks—joint classification and feature alignment—are key innovations that improve recognition accuracy without adding inference overhead. This approach bridges the gap between pose-based and appearance-based models, setting a new standard for efficient multi-modal activity recognition.
Novelty
This is the first work to embed pose-guided auxiliary tasks into a video transformer for ADL recognition, achieving high accuracy without pose input at inference. Unlike prior methods that rely on pose sensors or computationally expensive pose estimation during inference, PI-ViT trains with pose supervision and discards it, making the model lightweight and scalable. Its innovative use of auxiliary tasks for pose induction during training distinguishes it from existing multi-modal approaches, offering a new paradigm for efficient, high-performance activity recognition.
Limitations
- The approach depends on accurate pose detection during training; errors in pose estimation can affect the quality of learned features. In scenarios with occlusion or poor pose detection, performance may degrade.
- Training complexity increases due to auxiliary tasks, requiring additional annotations and computational resources.
- The method's effectiveness in outdoor or highly cluttered environments remains to be validated, as current datasets are mostly indoor and controlled.
Future Work
Future research could explore self-supervised pose learning to reduce reliance on annotated skeleton data, extend the framework to outdoor and cluttered scenes, and optimize the modules for real-time deployment. Integrating adaptive mechanisms for dynamic auxiliary task weighting and exploring multi-task learning with other modalities (e.g., audio, depth) could further improve robustness and applicability across diverse scenarios.
AI Executive Summary
Video transformers have revolutionized human activity recognition, yet their reliance solely on RGB data limits their effectiveness in complex, real-world scenarios involving subtle or multi-view actions. Recognizing activities of daily living (ADL) presents unique challenges—actions are often visually similar, and viewpoints vary significantly. Existing models struggle to distinguish such nuances, especially without multi-modal cues.
This paper introduces PI-ViT, a novel approach that leverages human pose information during training to enhance RGB-based video transformers. The core innovation lies in two plug-in modules—2D Skeleton Induction Module and 3D Skeleton Induction Module—that perform auxiliary pose-aware tasks during training. These modules guide the model to learn pose-augmented features, improving fine-grained and view-invariant action recognition. Importantly, these modules are discarded during inference, ensuring no additional computational burden.
Experimental results demonstrate that PI-ViT achieves state-of-the-art performance on three major ADL datasets—Toyota-Smarthome, NTU120, and NTU60—outperforming existing methods by significant margins. On Toyota-Smarthome, it improves cross-view accuracy by nearly 7%, validating its robustness in multi-view scenarios. Its ability to recognize subtle actions, even without pose input at inference, marks a substantial advancement in practical activity understanding.
This work addresses key limitations of prior models—costly multi-modal inference and poor fine-grained discrimination—by integrating pose information during training only. The approach not only boosts accuracy but also maintains efficiency, making it suitable for real-world deployment in surveillance, eldercare, and assistive robotics. Future directions include self-supervised pose learning, outdoor scene adaptation, and real-time implementation, promising broader impact across AI-driven activity analysis.
Deep Analysis
Background
视频变换器在动作识别中表现出色，但多依赖RGB信息，受限于细粒度动作区分和多视角识别的挑战。早期方法如TimeSformer、VideoSwin在大规模数据集上取得突破，但在细节捕获和视角不变性方面仍有不足。骨架信息的引入，如OpenPose和Hyperformer，增强了运动和姿态理解，但在推理时依赖传感器或复杂预处理，限制了实际应用。近年来，结合多模态信息的研究不断推进，但多在训练或推理阶段依赖传感器，限制了应用范围。本文旨在解决这些瓶颈，提出无需额外推理成本的训练引导方法，提升日常活动识别的实用性。
Core Problem
核心问题在于如何在保持模型高效的同时，增强对细粒度动作和多视角变化的识别能力。现有RGB模型难以区分相似动作，且多视角识别受外观变化影响大。引入骨架信息虽有效，但在推理中依赖传感器或复杂预处理，限制了实际应用。如何在训练中利用骨架信息优化RGB特征，同时在推理时无需骨架输入，是当前的技术难点。解决该问题，将极大推动智能监控、健康监测等场景的应用落地。
Innovation
创新点在于引入两个辅助模块（2D-SIM和3D-SIM）在训练阶段引导模型学习姿态信息，提升细粒度和视角不变的动作识别能力。不同于传统多模态融合在推理时依赖骨架，本文通过辅助任务实现姿态引导，训练后剔除模块，减少推理成本。采用特定的姿态映射和特征对齐机制，增强RGB特征的表达能力，提升模型泛化能力。这一策略突破了多模态融合的成本瓶颈，为视频变换器在日常场景中的应用提供了新路径。
Methodology
- �� 输入：视频帧和人体骨架（训练阶段引入）
- �� 设计两个插件模块：2D-SIM通过映射骨架关节增强细节，3D-SIM通过运动特征对齐实现视角不变
- �� 训练过程中，辅助任务引导模型学习姿态信息，优化RGB特征
- �� 训练结束后，剔除插件模块，模型可在无骨架输入下进行推理
- �� 采用TimeSformer作为基础架构，插入两个插件模块
- �� 损失函数结合分类损失和辅助对齐损失，确保特征优化
- �� 在多个ADL数据集上进行评估，验证模型性能
Experiments
采用Toyota-Smarthome、NTU120和NTU60三大数据集，比较PI-ViT与现有SOTA方法。训练中使用Kinetics-400和Something-Something-v2预训练，指标包括准确率和跨视角性能。通过消融实验验证插件模块的贡献，调整位置和参数设置，确保模型在细粒度和视角不变识别中的优势。对比不同的辅助任务和特征对齐策略，分析模型的鲁棒性和泛化能力。
Results
PI-ViT在Toyota-Smarthome跨视角协议中提升6.9%，在NTU120和NTU60上分别达到95.1%和97.0%的最高准确率。引入骨架信息显著改善细粒度动作区分，验证了多模态引导的有效性。模型在推理阶段无需骨架输入，保持高性能，验证了训练引导的泛化能力。与传统多模态方法相比，PI-ViT在效率和准确性上均优越，展示了其在实际应用中的潜力。
Applications
该模型适用于智能监控、老年人辅助、家庭机器人等场景，能在无需额外传感器的情况下实现高效动作识别。只需视频输入，无需骨架检测，降低硬件成本，提升系统鲁棒性。未来可结合边缘计算，实现实时监控和交互，为智能家居和安防行业带来革命性变革。
Limitations & Outlook
模型在极端遮挡或骨架检测不准时表现受限，训练依赖大量标注数据，泛化到户外复杂环境仍具挑战。骨架引导机制增加训练复杂度，推理时无骨架信息可能影响部分微妙动作的识别。未来需优化骨架检测鲁棒性和模型自适应能力，以拓展应用范围。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多不同的机器，每台机器都有自己的特定动作。以前，工人只看机器的外表（颜色、形状）来判断它们在做什么，但有时候不同的机器长得很像，难以区分。现在，工厂引入了一个新系统，不仅看外表，还会用一些“骨架”信息——就像用骨架图来标记每台机器的关键部件。这些骨架信息帮助工人更快、更准确地知道机器在做什么，即使它们长得很像或从不同角度看。这个新系统在训练时学习了这些骨架，但在实际操作中，不需要再看骨架图，只用外表就能判断。这样既节省了时间，又提高了准确率。它就像教会工人用骨架图记忆机器的动作，然后让他们只看外表也能认出动作。这个方法让工厂的工作变得更智能、更高效，也可以用在监控视频、家庭机器人等地方，让机器更懂人类的动作和意图。
ELI14 Explained like you're 14
想象你在学校的操场上玩游戏，有很多人动作很像，比如两个朋友都在挥手，但你不太确定是谁。以前，可能只看他们的衣服或站的位置，但这样很容易搞错。现在，假设你还可以看到他们的骨架——像画骨架的线条，告诉你每个人的手、脚、头的位置。这样一看，就能更清楚是谁在做什么，即使他们穿着一样或从不同角度看。这个新方法就像给电脑也装上了“骨架线条”，让它学会用骨架判断动作。训练时，电脑会学习这些骨架信息，帮它更懂动作，但在真正用的时候，它只用普通的视频就能识别，不需要骨架线条。这就像你学会了用骨架线条记动作，之后不用看骨架也能认出动作。这样，电脑就变得更聪明、更快，可以帮忙监控、照顾老人，甚至帮机器人更好地理解人类的动作。是不是很酷？
Abstract
Video transformers have become the de facto standard for human action recognition, yet their exclusive reliance on the RGB modality still limits their adoption in certain domains. One such domain is Activities of Daily Living (ADL), where RGB alone is not sufficient to distinguish between visually similar actions, or actions observed from multiple viewpoints. To facilitate the adoption of video transformers for ADL, we hypothesize that the augmentation of RGB with human pose information, known for its sensitivity to fine-grained motion and multiple viewpoints, is essential. Consequently, we introduce the first Pose Induced Video Transformer: PI-ViT (or $π$-ViT), a novel approach that augments the RGB representations learned by video transformers with 2D and 3D pose information. The key elements of $π$-ViT are two plug-in modules, 2D Skeleton Induction Module and 3D Skeleton Induction Module, that are responsible for inducing 2D and 3D pose information into the RGB representations. These modules operate by performing pose-aware auxiliary tasks, a design choice that allows $π$-ViT to discard the modules during inference. Notably, $π$-ViT achieves the state-of-the-art performance on three prominent ADL datasets, encompassing both real-world and large-scale RGB-D datasets, without requiring poses or additional computational overhead at inference.