Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
Proposes Skeleton-to-Image encoding (S2I) enabling self-supervised skeleton representation learning via vision-pretrained models, improving cross-format generalization.
Key Findings
Methodology
This work introduces S2I, which partitions 3D skeleton joints into five semantic body parts, sorts joints within each part, stacks joint coordinates over time, and interpolates to 224×224 pixels, forming an image-like tensor. This encoding allows direct use of large-scale pretrained vision models like MAE and DiffMAE for self-supervised skeleton pretraining. The approach eliminates the need for task-specific skeleton architectures and supports heterogeneous skeleton formats, facilitating cross-dataset and cross-format transfer learning. Experiments on NTU-60, NTU-120, and PKU-MMD datasets demonstrate the method’s robustness and effectiveness in both in-domain and cross-domain scenarios.
Key Results
- On NTU-60 C-sub, the image pretraining baseline achieved 52.0% accuracy with linear probing and 82.8% after fine-tuning. Skeleton pretraining with MAE improved performance to 90.4%. Cross-format evaluation showed superior transferability, especially in heterogeneous skeleton layouts. Similar trends were observed on NTU-120 and PKU-MMD, with significant gains in robustness and generalization.
- In multi-modal experiments, S2I combined with pretrained models enhanced action recognition accuracy, outperforming traditional skeleton models. Ablation studies confirmed that joint, motion, and skeleton modality choices impact performance, with joint-based masking strategies yielding the best results.
- The results validate that S2I effectively bridges skeleton data with vision models, enabling universal, scalable, and format-agnostic skeleton representation learning with high transferability.
Significance
This approach fundamentally shifts skeleton action recognition by leveraging the power of large-scale vision pretraining, addressing the challenge of data heterogeneity and scarcity. It paves the way for universal skeleton representations, reducing dependence on dataset-specific architectures and annotations. The method’s robustness across formats and datasets opens new avenues for multi-modal, real-world applications such as surveillance, human-computer interaction, and AR/VR, where diverse skeleton data sources are common. It also fosters integration of skeleton data with other visual modalities, advancing the field toward more flexible and scalable models.
Technical Contribution
The core technical innovation is the S2I encoding, transforming sparse, articulated skeleton sequences into dense, image-like tensors compatible with vision transformers. This enables direct transfer of pretrained weights, facilitating self-supervised learning without designing new skeleton-specific architectures. The framework supports heterogeneous data formats, improves cross-dataset transfer, and integrates seamlessly with existing vision models like MAE and DiffMAE. The approach also introduces novel masking strategies tailored for skeleton data, enhancing the learning of articulated and temporal dependencies.
Novelty
This is the first work to convert skeleton sequences into dense image-like representations for leveraging large-scale vision pretraining. Unlike prior methods that depend on fixed joint configurations or graph structures, S2I provides a format-agnostic, scalable interface supporting heterogeneous skeleton data. It enables direct application of masked autoencoding techniques, significantly improving transferability and generalization across diverse datasets and formats, marking a new paradigm in skeleton representation learning.
Limitations
- The method relies on semantic joint partitioning, which may be sensitive to extreme poses or occlusions, affecting robustness.
- Interpolation might introduce artifacts, potentially impacting fine-grained action recognition.
- Performance in multi-person scenarios or highly complex actions remains to be validated, requiring further adaptation.
Future Work
Future directions include integrating multi-modal data such as RGB and depth to enhance recognition robustness, optimizing joint partitioning and interpolation algorithms, and exploring generative models to enrich skeleton representations. Extending the approach to real-time applications and multi-person scenarios will also be key research areas.
AI Executive Summary
Recent advances in large-scale vision pretraining models like MAE and DiffMAE have revolutionized visual representation learning, yet their application to 3D skeleton data remains limited due to structural differences. Traditional skeleton-based action recognition methods often depend on fixed joint configurations and dataset-specific architectures, hindering scalability and cross-dataset generalization. To bridge this gap, this work introduces Skeleton-to-Image encoding (S2I), a novel approach that transforms sparse skeleton sequences into dense, image-like tensors. This transformation involves partitioning joints into five semantic body parts, sorting joints within each part, stacking joint coordinates over time, and interpolating to a standard 224×224 size. The resulting representation enables direct utilization of pretrained vision models, such as MAE and DiffMAE, for self-supervised skeleton pretraining, effectively transferring visual domain knowledge to skeleton analysis. Extensive experiments on NTU-60, NTU-120, and PKU-MMD datasets demonstrate that S2I significantly improves transferability, especially in cross-format and cross-dataset scenarios, outperforming traditional methods. The approach’s format-agnostic nature allows it to handle heterogeneous skeleton data seamlessly, fostering universal skeleton representation learning. This innovation opens new avenues for multi-modal action recognition, human-computer interaction, and surveillance, reducing the reliance on large labeled skeleton datasets. Despite its strengths, challenges remain in handling occlusions, extreme poses, and multi-person scenarios. Future work will focus on integrating multi-modal data, refining joint partitioning strategies, and extending real-time capabilities. Overall, S2I marks a substantial step toward scalable, flexible, and knowledge-rich skeleton understanding, leveraging the power of vision pretraining in a novel way.
Deep Analysis
Background
人体动作识别是计算机视觉中的核心任务之一，经过多年的发展，从早期的手工特征到深度学习模型不断演进。早期方法如RNN、CNN和图卷积网络（GCN）在骨架数据建模方面取得一定成果，但存在对骨架格式依赖强、泛化能力有限的问题。近年来，Transformer架构的引入显著提升了长距离依赖建模能力，代表性工作如ST-GCN、MS-G3D和骨架Transformer等，强调结构化建模和多模态融合。然而，这些方法普遍依赖于固定的关节定义和骨架格式，难以应对多源异构数据。与此同时，预训练模型如ViT和MAE在图像任务中表现优异，但缺乏有效的骨架数据适配策略，导致其在骨架领域的应用受限。解决这一瓶颈，成为推动骨架动作识别普及和应用的关键。
Core Problem
现有骨架动作识别方法多依赖于特定骨架格式，导致模型难以跨数据源迁移。骨架数据的稀疏性和结构多样性，使得模型在面对不同采集设备或关节定义时表现不佳。此外，缺乏大规模标注数据限制了模型的泛化能力。如何将丰富的视觉预训练知识迁移到骨架领域，成为亟需解决的问题。传统方法多设计专用骨架网络，缺乏通用性和扩展性，限制了多源、多模态融合的潜力。
Innovation
本研究的核心创新在于提出Skeleton-to-Image编码（S2I），将稀疏骨架序列转化为密集的图像样式数据，兼容大规模预训练视觉模型。具体包括：• 关节分割：将骨架划分为五个身体部分，保持结构关系；• 空间-时间堆叠：沿时间维度堆叠关节坐标，形成空间-时间特征图；• 图像插值：将空间和时间维度线性插值至224×224像素，确保模型输入一致性。该方法无需设计专用骨架模型，便于跨格式、多源数据的统一预训练。结合MAE和DiffMAE进行自监督学习，有效提升骨架表示的迁移能力。此创新突破了传统骨架模型的局限，为多模态融合和大规模预训练提供了新途径。
Methodology
- �� 输入：原始3D骨架序列（T×J×3）；• 分割：将关节划分为五个身体部分，按结构排序；• 空间堆叠：沿时间维度堆叠关节坐标，形成T×J×3的空间-时间特征图；• 转换：将x、y、z坐标映射到RGB通道，形成T×J×3的伪图像；• 插值：对空间和时间维度进行线性插值，调整至224×224像素；• 预训练：利用MAE、DiffMAE等模型进行自监督预训练；• 迁移：在下游动作识别任务中进行微调或线性探测。
Experiments
在NTU-60、NTU-120和PKU-MMD数据集上，采用标准交叉主体和交叉视角协议，比较不同预训练策略和骨架格式的迁移效果。实验中，使用ViT-B架构，调节掩码比例、骨架分割策略，验证不同设置对性能的影响。通过消融分析，确认S2I在跨格式和异构数据场景中的优势。模型性能以准确率和F1-score衡量，结果显示预训练模型在迁移学习中表现优异，显著优于传统骨架模型。
Results
S2I结合预训练模型在NTU-60 C-sub任务中，线性探测达52.0%，微调后提升至82.8%；骨架预训练模型（如MAE）在同任务中达到90.4%。跨格式评估中，S2I表现出优越的鲁棒性，尤其在不同骨架布局和数据源间迁移效果显著优于传统方法。多模态融合实验显示，S2I能有效增强多模态动作识别性能，验证了其广泛适用性。
Applications
该技术适用于智能监控、虚拟现实、人体动作分析等场景，特别是在多源、多模态数据融合需求强烈的应用中。通过统一的表示方式，简化了模型设计和数据预处理流程，降低了部署难度。未来结合深度学习和生成模型，有望实现更高效的动作理解和交互体验。
Limitations & Outlook
当前方法依赖于关节语义划分，可能在极端姿态或遮挡情况下表现不佳。插值处理可能引入信息损失，影响细粒度识别。模型在复杂多人体场景中的泛化能力尚需验证。未来需优化骨架分割策略和插值算法，提升鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在厨房做饭，所有食材都放在不同的碗里。每次做菜，你需要把食材按照一定顺序放好，比如先放蔬菜，再放肉，最后放调料。现在，如果你有很多不同的厨房，每个厨房的食材摆放方式都不一样，但你还是希望用同样的菜谱做菜。这个研究就像是把不同厨房的食材都整理成一样的“菜谱卡片”，让你不用担心厨房的不同，只要看这张卡片就知道怎么做菜。它把复杂的骨架动作变成了简单的图片，让电脑也能像看图片一样理解动作。这样，无论骨架数据来自哪个设备或格式，都可以用一样的方法学习和识别动作，就像用同一份菜谱做不同厨房的菜一样方便。
ELI14 Explained like you're 14
想象你在玩一个游戏，你的角色可以做各种动作，比如跳跃、跑步、挥手，但每次游戏里的动作都用不同的方式描述。有的用骨架线条，有的用点点，格式都不一样。这个研究就像是发明了一种神奇的相机，把所有不同的动作都拍成一张漂亮的照片，不管它们来自哪个游戏或哪个设备。这样，电脑就可以用它学会的动作识别出各种动作，不用担心格式不同的问题。研究还发现，把骨架动作变成图片后，电脑用它学会的动作比以前更快、更准，就像用一张万能的照片识别各种动作一样厉害。这项技术可以帮助监控系统更聪明，甚至让虚拟人物更自然地模仿人类动作。未来，这种方法还能让不同设备之间更好地交流，带来更酷的虚拟现实体验！
Abstract
Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including cross-modal and multi-modal scenarios. However, their direct application to 3D human skeleton data remains challenging due to fundamental differences in data format. Moreover, the scarcity of large-scale skeleton datasets and the need to incorporate skeleton data into multi-modal action recognition without introducing additional model branches present significant research opportunities. To address these challenges, we introduce Skeleton-to-Image Encoding (S2I), a novel representation that transforms skeleton sequences into image-like data by partitioning and arranging joints based on body-part semantics and resizing to standardized image dimensions. This encoding enables, for the first time, the use of powerful vision-pretrained models for self-supervised skeleton representation learning, effectively transferring rich visual-domain knowledge to skeleton analysis. While existing skeleton methods often design models tailored to specific, homogeneous skeleton formats, they overlook the structural heterogeneity that naturally arises from diverse data sources. In contrast, our S2I representation offers a unified image-like format that naturally accommodates heterogeneous skeleton data. Extensive experiments on NTU-60, NTU-120, and PKU-MMD demonstrate the effectiveness and generalizability of our method for self-supervised skeleton representation learning, including under challenging cross-format evaluation settings.