Towards Surgical World-Action Modeling: A Preliminary Joint Visual-Trajectory Forecasting for Surgical Motion Planning

The evolution of surgical robotics has driven research toward scene understanding and motion prediction. Early efforts focused on trajectory modeling (e.g., Hansen et al. 2026) or visual scene synthesis (e.g., Chen et al. 2025). While these approaches achieved progress in their domains, they lacked integration, limiting the ability to simulate realistic, dynamic surgical environments. Recent advances utilize deep learning, such as transformer-based models and diffusion techniques, to generate plausible videos (e.g., Surgsora 2025). However, these models often produce visually convincing scenes without guaranteeing geometric accuracy of instrument motion, or vice versa. The challenge remains to develop models that can jointly predict both visual evolution and instrument trajectories over long horizons, enabling more reliable and interpretable surgical planning. This gap motivates the current study, aiming to unify scene and motion modeling within a single predictive framework.

The central challenge is to accurately forecast both the visual state of the surgical scene and the instrument trajectories over extended periods. Existing models either focus on visual realism, risking inconsistency with actual instrument motion, or on trajectory accuracy, neglecting visual plausibility. Long-horizon predictions tend to suffer from error accumulation, leading to degraded scene quality and inaccurate instrument paths. This discrepancy hampers the deployment of autonomous surgical systems, which require stable, coherent predictions to ensure safety and effectiveness. The core difficulty lies in effectively modeling the complex, multimodal dynamics of surgical scenes, where tissue deformation, occlusion, and instrument-tissue interactions introduce high variability. Achieving joint, stable, and long-term predictions remains a pressing problem.

The key innovations include: 1) a joint latent encoding scheme that fuses visual and trajectory information into a shared representation, enabling multimodal understanding; 2) a spatiotemporal attention encoder that captures dependencies across time and space, improving context modeling; 3) a chunked autoregressive prediction strategy that decomposes long sequences into manageable segments, reducing error propagation; 4) the integration of scheduled sampling to mitigate training-inference mismatch. These innovations collectively advance the state-of-the-art by enabling stable, long-horizon joint predictions, addressing the limitations of prior single-task or single-modal models, and providing a more comprehensive understanding of surgical scene dynamics.

• �� Input: historical video frames I1:K and instrument trajectories P1:K. • Visual encoding: Use a pre-trained SurgMotion ViT-L encoder to extract intermediate and final visual tokens. • Trajectory embedding: Embed 2D instrument points via MLP. • Fusion: Combine visual tokens and trajectory embeddings into shared latent features. • Spatiotemporal encoding: Apply attention mechanisms to model dependencies, producing a context vector C. • Prediction heads: Separate modules predict residual visual states and trajectory corrections for each future step. • Chunked autoregression: Predict 3 steps at a time, update history, and repeat until 15 steps are forecasted. • Scheduled sampling: During training, probabilistically replace ground truth with model predictions to improve stability.

The model is trained and evaluated on SurgWMBench, which contains annotated endoscopic videos and instrument trajectories. The first five frames serve as context, predicting the next 15 frames. Metrics include PSNR, SSIM, LPIPS for visual fidelity, and ADE, FDE for trajectory accuracy, evaluated across five segments. The training employs AdamW optimizer, with hyperparameters tuned for stability. Ablation studies compare single-step versus chunked prediction, demonstrating the latter’s superiority. The effectiveness of scheduled sampling is validated by improved long-term stability. Results show consistent performance gains, with PSNR reaching 23.11dB and trajectory ADE halving, confirming the approach’s robustness.

Quantitative analysis reveals that chunked autoregressive prediction significantly outperforms direct one-shot methods across all horizons. The first segment PSNR improves from 18.86dB to 23.11dB, and trajectory ADE reduces from 45.77 to 22.22 pixels. The model maintains better visual and motion fidelity up to the 15th step, though some degradation occurs beyond the 12th step. The scheduled sampling technique effectively reduces error accumulation, leading to more stable long-horizon forecasts. These findings validate the hypothesis that decomposing long sequences into shorter segments enhances prediction quality and stability.

This joint visual-trajectory forecasting framework can be integrated into autonomous surgical robots, providing real-time scene understanding and motion planning. It can also serve as a training tool for surgeons, simulating realistic surgical scenarios. Additionally, it offers a foundation for developing more sophisticated decision support systems that anticipate surgical scene evolution, thereby improving safety and efficiency in complex procedures.

Despite promising results, the model's predictions degrade over long horizons due to error accumulation and visual artifacts. It relies heavily on high-quality annotated datasets, limiting generalization to diverse surgical environments. The current latent representations are not directly reconstructed into pixel-level images, affecting visual interpretability. Future work should incorporate uncertainty modeling, action conditioning, and end-to-end pixel synthesis to address these issues and improve robustness in real-world applications.