Reconstructing Hands in 3D with Transformers
HaMeR employs a Transformer-based architecture with large-scale data, achieving state-of-the-art 3D hand mesh reconstruction from monocular images, outperforming previous methods.
Key Findings
Methodology
HaMeR utilizes a fully transformer-based framework, integrating a large Vision Transformer (ViT-H) backbone with a transformer decoder to regress MANO hand model parameters and camera pose. It combines multi-source datasets, including controlled 3D annotations (FreiHAND, HO3Dv2) and in-the-wild 2D keypoints (COCO, HInt). The training employs a combination of 3D losses, reprojection losses, and adversarial training to improve robustness. The model is trained on over 2.7 million examples, enabling it to generalize across diverse scenarios. Evaluation on standard benchmarks shows significant improvements, with PA-MPJPE around 6mm on FreiHAND and 7.7mm on HO3Dv2, surpassing prior methods. The newly annotated HInt dataset further demonstrates the model’s effectiveness in unconstrained environments, with [email protected] exceeding 85%.
Key Results
- On FreiHAND, PA-MPJPE reached 6.0mm, outperforming MeshGraphormer and MobRecon, with consistent gains across metrics.
- On HO3Dv2, PA-MPJPE was 7.7mm, showing superior accuracy and robustness.
- On the HInt dataset, [email protected] exceeded 85% across various in-the-wild scenarios, especially under occlusion and interaction conditions, validating model generalization.
Significance
This work advances monocular 3D hand reconstruction by leveraging large-scale data and Transformer architectures, addressing the challenge of robustness in complex, real-world environments. It bridges the gap between controlled benchmarks and practical applications like AR/VR, gesture control, and human-computer interaction, providing a scalable solution that can operate reliably in diverse scenarios. The introduction of the HInt dataset further enriches the evaluation landscape, fostering future research in unconstrained settings. Overall, it marks a significant step toward deploying 3D hand pose estimation in real-world applications, with broad implications for industry and academia.
Technical Contribution
The paper introduces a novel Transformer-based architecture, combining a large Vision Transformer backbone with a transformer decoder for joint regression of hand shape, pose, and camera parameters. It integrates multi-source large-scale datasets, employs multi-task and adversarial losses, and constructs a new in-the-wild dataset, HInt, annotated with 2D keypoints and occlusion labels. These innovations collectively improve accuracy, robustness, and generalization, setting new benchmarks in monocular 3D hand mesh recovery.
Novelty
This is the first application of large-scale Vision Transformers for 3D hand mesh reconstruction from monocular images. The combination of multi-source data fusion, adversarial training, and in-the-wild dataset annotation distinguishes this work from prior parametric or non-parametric methods, significantly enhancing robustness and accuracy in unconstrained environments.
Limitations
- Despite improvements, the model still struggles with extreme occlusions and highly complex hand-object interactions, indicating room for further robustness enhancement.
- Training requires extensive annotated datasets, which are costly and time-consuming to produce, limiting scalability.
- Inference speed remains relatively slow for real-time applications, necessitating optimization.
Future Work
Future directions include model compression and acceleration for real-time deployment, integrating multi-modal data such as depth or tactile information, and exploring self-supervised learning to reduce dependency on large annotated datasets. Expanding the dataset to cover more diverse environments and interactions will further improve robustness. Additionally, integrating temporal modeling for video sequences could enhance temporal consistency and dynamic tracking capabilities.
AI Executive Summary
Understanding and reconstructing human hands in 3D from monocular images is a long-standing challenge in computer vision, vital for applications ranging from virtual reality to human-computer interaction. Existing methods often struggle with accuracy and robustness, especially under occlusion, diverse viewpoints, and complex interactions. Addressing these issues, the present work introduces HaMeR, a Transformer-based model that leverages large-scale data and advanced neural architectures to achieve unprecedented performance.
HaMeR’s core innovation lies in adopting a large Vision Transformer (ViT-H) backbone combined with a transformer decoder, enabling the model to effectively learn rich representations of hand pose and shape. The training process integrates multiple datasets, including controlled 3D annotations (FreiHAND, HO3Dv2) and in-the-wild 2D keypoints (COCO, HInt). The loss functions encompass 3D parameter regression, reprojection consistency, and adversarial discriminators, collectively enhancing robustness against occlusion and complex interactions.
Experimental results demonstrate that HaMeR surpasses previous state-of-the-art methods, achieving a PA-MPJPE of 6.0mm on FreiHAND and 7.7mm on HO3Dv2, with significant improvements in in-the-wild scenarios. The newly annotated HInt dataset, covering diverse real-world environments, shows [email protected] exceeding 85%, validating the model’s generalization. These advances open new avenues for deploying accurate, reliable hand tracking in practical applications like AR/VR, sign language recognition, and robotics.
Despite these achievements, challenges remain in handling extreme occlusions and reducing inference latency. Future work will focus on model compression, multi-modal data integration, and temporal modeling for video sequences. Overall, HaMeR marks a major step forward in monocular 3D hand reconstruction, bridging the gap between laboratory research and real-world deployment, and setting a new benchmark for future innovations.
Deep Dive
Abstract
We present an approach that can reconstruct hands in 3D from monocular input. Our approach for Hand Mesh Recovery, HaMeR, follows a fully transformer-based architecture and can analyze hands with significantly increased accuracy and robustness compared to previous work. The key to HaMeR's success lies in scaling up both the data used for training and the capacity of the deep network for hand reconstruction. For training data, we combine multiple datasets that contain 2D or 3D hand annotations. For the deep model, we use a large scale Vision Transformer architecture. Our final model consistently outperforms the previous baselines on popular 3D hand pose benchmarks. To further evaluate the effect of our design in non-controlled settings, we annotate existing in-the-wild datasets with 2D hand keypoint annotations. On this newly collected dataset of annotations, HInt, we demonstrate significant improvements over existing baselines. We make our code, data and models available on the project website: https://geopavlakos.github.io/hamer/.