Procedural Scene Programs for Open-Universe Scene Generation: LLM-Free Error Correction via Program Search

Scene synthesis has evolved from manual rule-based methods to data-driven neural models, such as Fisher et al.'s statistical layout models and Li et al.'s neural scene generators. While neural methods excel at generating realistic scenes within fixed categories, they lack flexibility for open-vocabulary, diverse environments. The advent of large language models (LLMs) like GPT-3 has opened new avenues, enabling text-driven scene generation. Early LLM-based approaches, such as LayoutGPT, used iterative coordinate prediction but faced issues with physical plausibility and scene coherence. Declarative methods introduced constraint-based programs to enforce relations, improving layout validity but suffering from computational inefficiency. This paper builds upon these developments by proposing an imperative, relation-centered program language combined with symbolic error correction, addressing the need for scalable, flexible, and human-aligned scene synthesis.

The main challenge is to generate diverse, physically plausible, and relation-consistent 3D scene layouts from open-ended text prompts efficiently. Existing methods either rely on slow constraint solvers, which hinder real-time editing and large-scale scene generation, or on neural predictors that often produce inconsistent or invalid layouts. The core bottleneck is balancing expressiveness, computational efficiency, and robustness. Additionally, ensuring that generated scenes align with human perceptual preferences remains difficult. Addressing these issues requires a framework capable of expressive scene description, fast error correction, and perceptually aligned evaluation, especially for complex, large-scale environments.

The paper's key innovations include: 1) the development of PSDL, a relation-centric scene description language supporting parametric relationships, local coordinate frames, and control flow, enabling expressive and compact scene representations; 2) a symbol-based, local search error correction mechanism that iteratively refines scene programs by fixing overlaps, out-of-bounds, and relation violations without additional LLM calls; 3) an evaluation protocol combining human preference tests and automated metrics, ensuring perceptual alignment. These innovations collectively improve the efficiency, robustness, and realism of scene layout generation, surpassing prior imperative and declarative methods in both quality and scalability.

• �� The pipeline begins with an LLM generating a scene template, including scene dimensions and object list.
• �� Next, the LLM writes a scene program in PSDL, defining object positions, relations, and parameters, utilizing control flow for pattern expressions.
• �� The symbolic error correction module evaluates the generated program by executing it, detecting violations like overlaps or out-of-bounds.
• �� Using a local search strategy, the module iteratively adjusts program parameters, preserving scene relations and improving layout validity.
• �� The process continues until the layout satisfies predefined criteria or reaches iteration limits.
• �� Finally, object retrieval modules fetch 3D meshes, completing the scene for visualization.

The authors evaluate their method across diverse scene categories, including indoor, outdoor, and highly structured environments like museums. They compare against baseline methods such as LayoutGPT and declarative constraint solvers, using human preference surveys and automated CLIP-based similarity metrics. The experiments measure layout validity, correction speed, and perceptual quality. Ablation studies examine the impact of relation modeling, parameter sharing, and control flow. Results show the proposed method achieves higher preference scores, better structural coherence, and faster correction times, especially in large, complex scenes, demonstrating its scalability and robustness.

The method's layouts are preferred in 82% and 94% of user tests over two declarative baselines, indicating superior perceptual quality. Automated metrics correlate strongly (>0.85) with human preferences across various scene types. The local search error correction reduces layout violations by over 30%, significantly improving physical plausibility. In large-scale environments like museums, the approach maintains scene diversity and coherence while decreasing correction time, validating its scalability. Ablation results confirm that parametric relationships and control flow are critical for performance, highlighting the importance of the proposed language features.

This framework enables rapid, flexible scene generation from natural language for virtual reality, gaming, and architectural visualization. It allows non-experts to create complex environments without extensive data or manual editing. Its efficiency makes it suitable for real-time scene editing and interactive design tools. Long-term, integrating learning-based relation discovery and multi-modal inputs could further automate and enhance scene synthesis, making it accessible for broader creative and industrial applications, including autonomous scene understanding and editing.

While effective, the approach faces challenges in handling scenes with highly complex or dynamic relations, where local search may converge to suboptimal solutions. Scalability remains an issue for extremely large environments with many objects and relations, requiring further optimization. Dependence on predefined relations and parametric structures may limit scene diversity, necessitating adaptive or learning-based relation discovery. Future work should address these limitations to realize fully autonomous, real-time, and highly diverse scene synthesis systems.