DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery

The evolution of egocentric hand tracking has transitioned from early detection-based methods like MANO regression [5, 20] to temporal fusion approaches [13, 17], addressing occlusion and motion challenges. Recent advances incorporate generative models such as ViDiHand [36], leveraging their ability to encode physical priors. Despite progress, these methods rely heavily on pixel-level sampling, calibration-dependent camera models, and struggle with long-term occlusion and out-of-view hands. The need for scalable, robust, and calibration-free solutions remains pressing, especially for real-world robotics and AR applications where occlusion and rapid movements are common.

The core challenge is reconstructing continuous, accurate 3D hand trajectories during occlusion and out-of-sight intervals in egocentric videos. Traditional detectors fail when hands are occluded or leave the frame, causing trajectory discontinuities. Windowed temporal models lack long-range reasoning, and calibration-dependent methods are fragile under camera motion. Overcoming these limitations requires a model capable of long-term reasoning, occlusion handling, and calibration-free inference, all while maintaining high accuracy in diverse, real-world scenarios.

This work introduces several key innovations: 1) transforming a pretrained diffusion model into a deterministic scene encoder, enabling efficient feature extraction; 2) employing clip-level features that encode physical and motion priors; 3) designing a bidirectional spatiotemporal decoder that leverages long-range attention for occlusion reasoning; 4) implementing a ray-based camera solver that predicts viewing directions without known intrinsics, supporting calibration-free inference. These innovations collectively enable robust, continuous hand tracking during severe occlusion, surpassing existing detection and generative methods.

• �� Encode input videos using a pretrained DiT diffusion transformer, extracting clip-level features via a single forward pass at zero noise. • Freeze a VAE encoder to compress videos into clean latent representations, which are processed by DiT to produce dense feature grids. • Construct a bidirectional decoder with spatially grounded queries for each frame, including hand, joint, and shape tokens, integrating positional and ray encodings. • Use cross-attention to extract per-joint 2D heatmaps and regress 3D pose parameters, orientations, and visibility scores. • Incorporate a shape head that pools tokens across the clip for consistent shape estimation. • Employ a ray head to predict viewing directions, enabling intrinsics-free depth and translation estimation via a PnP-like solver. • Train end-to-end with multi-task losses supervising pose, shape, visibility, and camera parameters, including out-of-sight hand supervision.

The model is trained on a mixture of egocentric datasets (ARCTIC, HOT3D, H2O, OakInk2) and rendered videos, evaluated on five benchmarks including ARCTIC, HOT3D, HOI4D, H2O, and OakInk2. Metrics include MPJPE, PA, detection F1, and jitter. Hyperparameters such as clip length, attention layers, and latent dimensions are tuned via ablation studies. The model's robustness is validated through comparisons with ten baselines, including detection-based and generative approaches, demonstrating superior performance especially in occlusion-heavy scenarios. Cross-dataset generalization and ablation studies confirm the importance of clip-level features, bidirectional reasoning, and ray-based localization.

On ARCTIC, MPJPE drops from 21.67mm to 15.26mm, a 30% improvement; HOT3D shows a reduction from 21.51mm to 12.89mm, a 40% gain; on HOI4D, error decreases by 23%. The detection F1 reaches 1.0, indicating near-perfect detection. The model maintains high continuity during hand occlusion and out-of-view periods, outperforming all baselines in accuracy, robustness, and temporal consistency. Ablation results highlight the contributions of clip-level features, bidirectional attention, and the ray-based solver in achieving these gains.

This technology can be deployed in immersive VR systems, robotic manipulation, and assistive devices, especially where occlusion and rapid motion are prevalent. Its calibration-free inference simplifies setup, reducing costs and complexity. The robustness to occlusion and out-of-sight scenarios makes it suitable for real-world applications like remote surgery, industrial automation, and AR/VR gaming. Future extensions could include multi-hand tracking, multi-object interaction, and integration with depth sensors for enhanced accuracy.

Despite its robustness, the model may still struggle with extremely fast or complex motions, or when hands are heavily occluded from unusual angles. Training requires large annotated datasets and significant computational resources, limiting real-time deployment currently. The inference speed needs further optimization for practical use, and generalization to diverse camera setups and environmental conditions remains an ongoing challenge. Future work should focus on model compression, multi-modal fusion, and domain adaptation to address these issues.