PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking

Point tracking has evolved from classical optical flow methods like Horn-Schunck and Lucas-Kanade to deep learning approaches such as FlowNet and RAFT, which significantly improved accuracy. However, most datasets focus on short sequences, limiting models’ ability to learn long-range temporal dependencies. Synthetic datasets like FlyingChairs, FlyingThings3D, and Kubric provided diversity but lacked scene complexity and realistic motion. Recent benchmarks like TAP-Vid emphasize multi-frame tracking but still fall short in scale and realism. The challenge remains to develop datasets and algorithms capable of handling long videos with occlusion, deformation, and scene interactions, essential for real-world applications like robotics and AR.

Existing point tracking methods are constrained by short temporal windows, typically 8 frames, which hampers their ability to handle occlusions longer than this span. They also rely heavily on initial appearance features, making them vulnerable to appearance changes over time. These limitations result in drift, reduced accuracy, and poor robustness in complex, dynamic scenes. Addressing these issues requires a dataset that captures long-term, realistic motion, and algorithms capable of leveraging extended temporal context while maintaining computational efficiency.

The paper introduces three key innovations: 1) PointOdyssey dataset, combining real motion capture, scene modeling, and SfM camera trajectories to generate diverse, realistic long videos; 2) PIPs++ tracker, replacing the MLP-Mixer with a deep 1D Resnet to extend temporal receptive fields up to 35 frames; 3) Multi-template mechanism, which adaptively updates appearance models using recent frames, improving robustness to appearance changes and occlusions. These innovations enable models to utilize longer temporal context, handle scene complexity, and improve long-term tracking accuracy.

• �� Data generation: Collect motion capture data for humans, animals, and scene environments; drive character animations with real motion data; simulate outdoor and indoor scenes with physics-based interactions; sample camera trajectories from SfM and manual paths; randomize scene attributes for diversity; render videos with atmospheric effects. • Annotation: Export 2D/3D point trajectories, occlusion, depth, normals, masks. • Model architecture: Replace PIPs’ MLP-Mixer with a 1D Resnet to process extended temporal sequences; incorporate multi-template features by sampling from recent frames; initialize with first-frame features; perform iterative updates with correlation and residual networks. • Training: Use the synthetic dataset, optimize with standard loss functions, validate on held-out sequences, and fine-tune for real-world benchmarks.

The models trained on PointOdyssey are evaluated on the dataset’s test split and real benchmarks (BADJA, TAP-Vid). Metrics include point tracking accuracy, trajectory length, and occlusion recovery. Ablation studies compare the impact of temporal window size, template updates, and scene randomization. Hyperparameters such as kernel size, number of templates, and training epochs are tuned. Results show that PIPs++ with extended receptive field and multi-template updates outperforms baseline PIPs and TAP-Net, especially during long occlusions and scene changes. Cross-scene tests confirm improved generalization, validating the dataset’s effectiveness.

The proposed method achieves 15% higher accuracy than original PIPs on PointOdyssey, with an average trajectory length of 2000 frames, surpassing prior datasets. It maintains stable tracking during occlusions exceeding 8 frames, with a 30% reduction in drift. On real-world datasets, success rates increase by 20%, demonstrating better scene understanding and robustness. Ablation confirms that extending temporal window and multi-template updates are critical for these gains. The dataset’s diversity and realism facilitate training models that generalize well across scenarios.

This work enables robust long-term tracking in applications like autonomous robots navigating complex environments, augmented reality systems maintaining scene consistency, and advanced video editing tools for object continuity. Its ability to handle occlusion, deformation, and scene interactions makes it suitable for real-time scene understanding, surveillance, and virtual production. The dataset and algorithms serve as a foundation for future research aiming to deploy long-term tracking in real-world dynamic systems.

Despite its realism, synthetic data may not capture all nuances of real-world scenes, affecting transferability. Computational costs for training and inference are high, especially for very long videos. Handling extreme occlusions, multi-object interactions, and scene semantics remains challenging, requiring further algorithmic sophistication and multi-modal data integration. Future work should focus on reducing computational load and improving scene understanding capabilities.