Inst4DGS: Instance-Decomposed 4D Gaussian Splatting with Multi-Video Label Permutation Learning
Inst4DGS introduces instance-decomposed 4D Gaussian splatting with differentiable Sinkhorn for multi-view consistent tracking, achieving PSNR 28.36 and mIoU 0.9129.
Key Findings
Methodology
This paper proposes Inst4DGS, which employs per-video latent permutation variables optimized via a differentiable Sinkhorn layer to align instance labels across multiple views. Combining this with an instance-decomposed motion scaffold, the approach models low-dimensional object trajectories for long-horizon optimization. The framework reconstructs dynamic scenes with high fidelity, supporting both tracking and segmentation tasks. Extensive experiments on Panoptic Studio and Neural3DV datasets demonstrate significant improvements in PSNR (from 26.10 to 28.36) and instance mIoU (from 0.6310 to 0.9129), outperforming state-of-the-art methods.
Key Results
- On Panoptic Studio, Inst4DGS achieves PSNR of 28.36, surpassing previous best of 26.10, and boosts instance mIoU from 0.6310 to 0.9129, indicating superior scene reconstruction and object identity preservation.
- The method effectively resolves cross-view label inconsistencies through learnable permutation latents, ensuring stable long-term object identities, validated by qualitative and quantitative results.
- The instance-decomposed motion scaffold enables efficient long-horizon trajectory optimization, reducing computational costs while maintaining accurate object motion modeling.
Significance
This work addresses the critical challenge of maintaining consistent object identities over long sequences in multi-view dynamic scenes. By integrating explicit label alignment with motion modeling, it advances scene understanding, enabling applications in AR/VR, autonomous driving, and robotics. The approach significantly improves reconstruction fidelity and tracking stability, paving the way for more reliable scene analysis in complex environments.
Technical Contribution
The core technical innovation lies in the integration of a differentiable Sinkhorn layer for cross-view label permutation learning, combined with an instance-decomposed motion scaffold for efficient trajectory optimization. This enables explicit, stable long-term tracking of multiple objects in dynamic scenes, supported by a novel 4D Gaussian representation that supports high-quality rendering and segmentation. The method bridges the gap between instance segmentation and long-term scene reconstruction, offering a scalable and robust framework.
Novelty
This is the first work to incorporate learnable, differentiable permutation matrices via Sinkhorn normalization into multi-view 4D Gaussian scene reconstruction, explicitly aligning instance labels across views. Coupled with an instance-decomposed motion scaffold, it provides a novel solution for long-term, identity-preserving tracking, setting a new benchmark in dynamic scene understanding.
Limitations
- Despite its robustness, the method's performance may degrade in scenarios with severe occlusion or extremely fast motion, where label matching becomes ambiguous.
- Training requires extensive multi-view segmentation data, which can be costly to obtain, limiting scalability in some real-world applications.
- Long-term trajectory optimization, while efficient, still faces challenges in highly cluttered or highly dynamic scenes, necessitating further robustness improvements.
Future Work
Future directions include reducing dependence on annotated segmentation data through self-supervised learning, enhancing label matching robustness, and extending the framework to real-time applications. Incorporating physical motion priors could further improve long-term trajectory stability, enabling broader deployment in autonomous systems.
AI Executive Summary
Inst4DGS represents a significant advancement in dynamic scene reconstruction and tracking, addressing the longstanding challenge of maintaining consistent object identities across multiple views over extended periods. By introducing a novel, differentiable Sinkhorn layer, the framework learns explicit cross-view instance label correspondences, effectively aligning identities despite inconsistent segmentation labels. This explicit label alignment, combined with an instance-decomposed motion scaffold, enables the model to optimize long-term object trajectories efficiently, supporting both high-fidelity rendering and precise segmentation.
The approach leverages a 4D Gaussian representation that models each object's spatial and temporal evolution, facilitating detailed scene understanding. Extensive experiments on the Panoptic Studio and Neural3DV datasets demonstrate that Inst4DGS outperforms existing methods, achieving PSNR of 28.36 and instance mIoU of 0.9129, surpassing prior state-of-the-art. The results highlight the method's ability to produce sharp boundaries, stable identities, and accurate long-term trajectories, making it highly suitable for applications in AR/VR, autonomous driving, and robotics.
This work not only advances the technical frontier of multi-view dynamic scene understanding but also opens new avenues for integrating explicit label supervision with trajectory modeling. Despite some limitations in highly cluttered or occluded scenarios, the framework's scalability and robustness mark a promising step toward real-world deployment. Future efforts will focus on reducing annotation costs, improving robustness, and enabling real-time processing, aiming to transform how machines perceive and interpret complex dynamic environments.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个大型工厂里，里面有许多机器人和工人在工作。每个人都在不断移动，但你只有几个摄像头拍到他们的动作。现在的问题是，如何知道每个人在不同时间、不同角度的动作是不是同一个人？传统的方法就像用放大镜盯着每个人，容易出错。这个新方法像给每个人贴上了特殊的标签，而且这些标签可以自己调整，让所有摄像头都认得出他们是谁。它还能记住每个人的跑动轨迹，画出一条长长的路径，不会把人搞丢。这样，工厂的管理系统就能更清楚地知道每个机器人和工人在干什么，协作也更顺畅。这个技术让复杂的场景变得像在工厂里追踪工人的动作一样简单。
ELI14 Explained like you're 14
想象你在学校操场上玩捉迷藏，有很多朋友在跑来跑去。你用几个不同的相机拍他们，但每个相机看到的名字都不一样。有时候，一个朋友在A相机叫“李明”，在B相机叫“张伟”，因为标签不一样。你想让所有相机都认得出他们是谁，这样就不会搞错。这个新方法就像给每个朋友贴上一个神奇的标签，这个标签可以自己调整，让所有相机都认得出他们。它还会记住每个人跑动的轨迹，画出一条长长的路径，不会把人搞丢。这样，不管看哪个相机，你都能知道每个朋友在干嘛、在哪儿。这个技术让我们更聪明地理解动态场景，就像在操场上找到每个朋友一样简单。
Abstract
We present Inst4DGS, an instance-decomposed 4D Gaussian Splatting (4DGS) approach with long-horizon per-Gaussian trajectories. While dynamic 4DGS has advanced rapidly, instance-decomposed 4DGS remains underexplored, largely due to the difficulty of associating inconsistent instance labels across independently segmented multi-view videos. We address this challenge by introducing per-video label-permutation latents that learn cross-video instance matches through a differentiable Sinkhorn layer, enabling direct multi-view supervision with consistent identity preservation. This explicit label alignment yields sharp decision boundaries and temporally stable identities without identity drift. To further improve efficiency, we propose instance-decomposed motion scaffolds that provide low-dimensional motion bases per object for long-horizon trajectory optimization. Experiments on Panoptic Studio and Neural3DV show that Inst4DGS jointly supports tracking and instance decomposition while achieving state-of-the-art rendering and segmentation quality. On the Panoptic Studio dataset, Inst4DGS improves PSNR from 26.10 to 28.36, and instance mIoU from 0.6310 to 0.9129, over the strongest baseline.