FF-ProCams: Feed-Forward Gaussian Splatting for Projector-Camera System

Projector-camera systems (ProCams) are vital for active scene perception and appearance control, widely used in spatial augmented reality and projection mapping. Traditional inverse rendering methods, though high in fidelity, are hampered by lengthy scene-specific optimization, making real-time applications impractical. Meanwhile, mainstream feed-forward 3D reconstruction models lack the ability to adapt to active projection illumination, limiting their effectiveness in dynamic environments. To bridge this gap, the authors propose FF-ProCams, a novel framework that combines a hybrid Mamba2-Transformer encoder with a relightable Gaussian primitive representation. This design enables the model to fuse multi-view geometric and photometric cues efficiently, predicting scene geometry and reflectance in a single forward pass. The introduction of a projector-aware differentiable renderer allows accurate simulation of observations under arbitrary projection patterns and poses, supporting relighting and pose variation without scene optimization. Training on a large synthetic dataset covering diverse geometries and materials, FF-ProCams demonstrates remarkable performance, reconstructing scenes from only 8 views with a speed of 0.13 seconds—over three orders of magnitude faster than traditional methods requiring hours. Experimental results show high fidelity in synthetic and real-world scenarios, with excellent generalization to unseen patterns and configurations. This breakthrough paves the way for real-time, interactive projection systems, significantly impacting AR, entertainment, and industrial applications. Despite these advances, challenges remain in modeling complex materials and handling calibration errors, guiding future research directions toward more robust, adaptive, and dynamic scene understanding solutions.