Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
Decoupled semantic-spatial Transformer improves NVS fidelity by separating features, achieving 1.1dB PSNR gain with minimal latency.
Key Findings
Methodology
The paper introduces a decoupled architecture using Independent-V attention, which maintains shared query-key routing for cross-branch interaction while keeping semantic and spatial tokens separate via independent value streams. It incorporates categorized supervision through iREPA with DINOv3 and geometric supervision via DA3 for the spatial branch. Bidirectional modulation enables dynamic cross-branch conditioning, enhancing feature clarity. Experiments on LVSM and Objaverse datasets demonstrate consistent improvements, with PSNR gains of over 1.1dB and negligible inference latency increase, validating the approach's effectiveness across different models.
Key Results
- On LVSM, the decoupled model achieves PSNR of 27.21, surpassing the baseline's 26.10, with SSIM reaching 0.869 and LPIPS dropping to 0.125. Incorporating supervision and modulation further boosts PSNR to 27.46. In Objaverse, PSNR reaches 26.46, showing robustness. Visual analysis reveals more structured, artifact-free features, confirming the method's ability to mitigate spatial bias interference. The model maintains real-time inference speed, suitable for practical deployment.
Significance
This work addresses a fundamental challenge in Transformer-based NVS—spatial bias interference—by proposing a novel decoupling strategy. It significantly enhances rendering fidelity, enabling more accurate and realistic view synthesis. The approach advances the theoretical understanding of feature disentanglement in high-dimensional models and offers practical benefits for real-time applications like VR, AR, and digital twins, pushing the frontier of scalable, high-quality 3D scene rendering.
Technical Contribution
The core innovation is the Independent-V attention mechanism, which decouples semantic and spatial tokens while sharing query-key routing for cross-branch interaction. The integration of categorized supervision via iREPA and geometric supervision via DA3 further refines feature representations. Bidirectional modulation allows dynamic, controlled feature interaction, leading to improved fidelity without additional inference cost. The architecture is compatible with various Transformer variants, broadening its applicability.
Novelty
This is the first work to fully decouple semantic and spatial features in feedforward Transformer-based NVS, employing shared Q/K routing with independent V streams. It introduces a bidirectional cross-branch modulation mechanism and leverages category and geometric supervision, effectively reducing spatial bias interference and artifacts, setting a new state-of-the-art in rendering quality.
Limitations
- The method relies on pre-trained category supervision, which may limit performance in unannotated or novel domains. High-resolution scenes still pose computational challenges, and extreme scene complexity can cause residual artifacts. The approach's effectiveness in dynamic or highly occluded scenes remains to be tested.
Future Work
Future directions include developing unsupervised or weakly supervised category learning methods, extending the framework to handle dynamic scenes, and optimizing for ultra-high-resolution rendering. Integrating multi-modal cues such as depth and semantics could further improve robustness and generalization, paving the way for truly scalable, real-time 3D scene synthesis.
AI Executive Summary
Novel View Synthesis (NVS) has become a pivotal task in computer vision, aiming to generate photorealistic new viewpoints from limited input images. Traditional methods like NeRF excel in quality but suffer from high computational costs and limited generalization. Transformer-based models, such as LVSM and GS-LRM, have introduced token-based architectures that encode semantic and geometric information jointly. However, this entanglement causes spatial bias to interfere with semantic appearance, leading to grid-like artifacts and reduced fidelity. To address this, the paper proposes a semantic-spatial decoupling approach using a novel Independent-V attention mechanism. This design maintains shared query-key routing for cross-branch interaction, while keeping semantic and spatial tokens separate via independent value streams, effectively mitigating spatial bias interference. The architecture is further enhanced with categorized supervision through iREPA with DINOv3 and geometric supervision via DA3, which guide the semantic and spatial branches respectively. Bidirectional modulation allows dynamic cross-branch conditioning, improving feature coherence and rendering quality. Extensive experiments on LVSM and Objaverse datasets demonstrate that the decoupled model achieves over 1.1dB PSNR improvement, with visual results showing sharper, more coherent structures and fewer artifacts. Importantly, this performance gain comes with almost no additional inference latency, making it suitable for real-time applications. The approach not only advances the state-of-the-art in NVS but also provides a new paradigm for disentangled, interpretable, and efficient 3D scene synthesis, with broad implications for virtual reality, AR, and digital twin industries. Future work will explore unsupervised learning, multi-modal integration, and high-resolution scaling, aiming to push the boundaries of real-time, high-fidelity view synthesis.
Deep Analysis
Background
随着虚拟现实、增强现实等技术的发展，新视角合成（NVS）成为研究热点。早期方法如NeRF通过体积渲染实现高质量视图，但计算成本高，泛化能力有限。Transformer架构引入后，LVSM、GS-LRM等模型采用Token机制，将语义信息（RGB块）与空间信息（Plücker射线）融合，提升效率。然而，空间信息的几何偏差与语义信息在统一空间中混合，导致特征模糊和伪影问题，限制了模型性能。近年来，特征表示的结构性与交互性成为焦点，试图解决空间偏差干扰语义的难题，推动高效、泛化的NVS发展。
Core Problem
现有Transformer模型在统一Token空间中，将空间几何信息与语义信息强制融合，导致空间偏差干扰语义表达，出现网格伪影和细节模糊。这严重影响渲染质量，限制模型在复杂场景中的应用。如何在保持表达能力的同时，将两类信息有效解耦，成为关键难题。解决方案需兼顾信息的独立表达和跨通道交互，确保模型既能利用空间几何信息，又能避免偏差干扰。
Innovation
提出语义-空间解耦Transformer，核心创新包括：1）V型注意力机制，保持Query-Key共享实现跨通道交互，同时Value流独立，确保信息纯粹。2）类别监督，通过iREPA与DINOv3结合，增强语义特征判别性。3）双向调制机制，实现空间与语义的动态互调，提升特征交互效率。这一设计突破了统一Token空间的限制，有效缓解空间偏差引起的伪影，提升渲染质量。
Methodology
- �� 输入：多视角图像与Plücker射线。• 分别用两个Tokenizer提取RGB块（语义）与Plücker射线（空间）。• Transformer中采用V型注意力机制：共享Q/K路由，独立V流，保持信息纯粹。• 引入类别监督，通过iREPA对语义Token进行对齐。• 利用DA3几何信息，监督空间Token的几何一致性。• 设计双向调制：空间调制语义，反之亦然，增强交互。• 训练目标结合RGB重建损失与类别、几何监督。• 推理时，模型几乎无延迟，性能优越。
Experiments
在RealEstate10K和Objaverse上进行评估，采用PSNR、SSIM、LPIPS指标。对比Entangled基线，解耦模型在PSNR提升1.1以上，SSIM达0.869，LPIPS降低至0.125。引入类别监督和调制后，性能再提升。通过消除空间偏差引起的伪影，验证了模型的鲁棒性。多模型、多数据验证其泛化能力，实验证明解耦设计在不同架构中均有效。
Results
解耦模型在LVSM上PSNR达27.21，优于Entangled模型的26.10，提升显著。引入类别监督与调制后，PSNR增至27.46，LPIPS降低至0.125。在Objaverse中，PSNR达26.46，表现优异。特征分析显示，结构更清晰，伪影明显减少，验证了空间偏差干扰的缓解效果。模型推理延迟几乎不变，适合实时应用。
Applications
该技术适用于虚拟现实、增强现实、数字孪生等场景，能实现高质量、实时的视图合成。对硬件要求较低，兼容多种Transformer架构，适合大规模部署。未来有望结合多模态信息，提升复杂场景下的表现，为行业带来革命性变革。
Limitations & Outlook
当前方法依赖预训练的类别监督，可能在未标注数据或新场景下表现不足。模型在极端复杂场景中仍存在细节模糊和伪影问题。解耦机制虽降低延迟，但在极端高分辨率下仍面临计算瓶颈。
Plain Language Accessible to non-experts
想象你在看一本画册，里面有很多图片，每张图片都代表一个场景。以前的模型像是用一块大布把所有信息都盖在一起，虽然可以看到整体，但细节容易模糊，还会出现一些奇怪的格子状伪影。现在的方法像是把画面中的内容分成两个部分：一部分是场景的“内容”——比如人物、物体的颜色和形状；另一部分是“位置”——比如物体在空间中的位置和角度。这样分开后，每个部分都能更清楚地表达自己，互相之间还能交流信息。这样做的好处是，画面变得更清晰，细节更丰富，也不会出现那些奇怪的格子。这个技术就像是给画册装上了两个不同的“眼睛”，让它们各自专注，又能合作，画出更漂亮的画面。
ELI14 Explained like you're 14
想象你在玩一个拼图游戏，拼图上的图片和拼图的形状其实是两个不同的东西。以前的拼图方法是把图片和形状混在一起，拼起来虽然能看出个大概，但细节会很模糊，还会出现一些奇怪的格子。现在的技术就像是把图片和形状拆开，分别用不同的工具拼，最后再把它们组合起来。这样一来，图片会变得更清楚，拼出来的画面也更逼真。就像你用两个不同的魔法棒，一个让图片变得更细腻，一个让空间位置更准确，最后合在一起，得到一个完美的画面。这个方法让虚拟世界变得更真实、更漂亮，也更快可以生成新视角的图片。
Glossary
V-shaped Attention (V型注意力)
一种在Transformer中实现语义与空间信息解耦的注意力结构，保持Query-Key共享，Value独立。
用于实现语义空间的解耦与交互。
Categorized Supervision (类别监督)
利用预训练模型提供的类别标签或特征对特定分支进行引导，提高特征判别能力。
增强语义Token的表达判别性。
Bidirectional Modulation (双向调制)
在两个信息通道间实现相互调节的机制，提升交互效率。
增强空间与语义之间的互调能力。
Plücker Rays (Plücker射线)
描述空间中射线的几何表示，携带空间结构信息。
作为空间信息输入，用于新视角合成。
Decoupled Tokens (解耦Token)
将语义信息与空间信息分别编码的Token，避免相互干扰。
核心创新点之一。
Open Questions Unanswered questions from this research
- 1 如何在无标注或弱监督环境中实现类别引导，提升模型的泛化能力仍未解决。
- 2 模型在极端复杂或高分辨率场景下的细节保持和伪影控制仍有提升空间。
Applications
Immediate Applications
Virtual Reality Content Generation
Enables real-time high-quality scene rendering, enhancing immersion for VR applications and virtual tours.
Long-term Vision
Digital Twins and Industrial Automation
Supports remote monitoring and virtual prototyping, enabling fully automated, intelligent scene synthesis.
Abstract
Transformer-based models have advanced feedforward novel view synthesis (NVS). Current architectures such as GS-LRM and LVSM mix semantic information (e.g., RGB) and spatial information (e.g., Plücker rays) into a shared feature space. Since Plücker rays naturally carry lattice-like spatial structure, these designs can make the spatial bias interfere with appearance representation and degrade rendering fidelity. To this end, we propose to decouple the representation of feedforward NVS transformers into separate semantic and spatial tokens. The decoupled design keeps semantic and spatial information explicit in their branches while preserving cross-branch interaction through shared attention routing. Built on this design, we introduce optional categorized supervision and bidirectional modulation: the former provides branch-specific training signals, while the latter improves interaction between the two branches. Notably, the base decoupled design introduces virtually zero additional inference latency due to its architectural design. The proposed designs achieve consistent improvements, demonstrating effectiveness across decoder-only and encoder-decoder feedforward NVS models.