VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement
VULCAN uses tool-augmented multi-agent collaboration with recursive backtracking to achieve robust multi-step 3D object arrangement, outperforming baselines.
Key Findings
Methodology
VULCAN integrates a multi-agent framework combining a planner, executor, and verifier. It employs MCP-based APIs for programmatic scene editing, enhancing visual grounding. Specialized visual tools analyze scene states, gather spatial info, and validate outcomes. The system uses a recursive backtracking search to efficiently explore multi-step arrangements, avoiding local optima. Each agent performs dedicated tasks: the planner formulates multi-step plans based on scene history, the executor applies API calls for object placement with collision avoidance, and the verifier assesses layout plausibility via visual feedback. This collaborative pipeline ensures high-fidelity, physically plausible scene rearrangement, with iterative refinement and error recovery.
Key Results
- In 25 complex 3D object arrangement tasks, VULCAN achieved an average collision rate of 4.6%, far below baseline methods (~45%). It scored 3.35/4 on layout plausibility and 3.55/4 on semantic alignment, outperforming single-step approaches. Ablation studies confirmed the importance of visual tools, backtracking, and multi-agent design. The system successfully decomposed complex instructions into multi-step plans with a success rate of 92%, significantly better than baselines.
- Across diverse instructions, VULCAN reliably generated coherent multi-step sequences, handling interdependencies and intermediate errors effectively. Human evaluations favored its results, citing higher physical correctness and visual plausibility. The approach demonstrated robust generalization across different scene complexities and instruction types, highlighting its practical potential for virtual scene design and robotic manipulation.
- The integration of external visual APIs, constraint-based optimization, and multi-agent cooperation enabled efficient scene analysis and layout generation. Recursive backtracking prevented dead-ends, ensuring goal achievement. The architecture supports complex, multi-object scenarios, setting a new standard for iterative scene manipulation in multimodal AI. The experimental results underscore the system’s superior performance and robustness.
Significance
This work advances the application of multimodal large language models in complex 3D scene manipulation, addressing key limitations of single-step and static methods. By introducing tool-augmented multi-agent collaboration, it significantly enhances robustness, scalability, and physical plausibility in multi-object arrangements. The approach bridges the gap between high-level language understanding and low-level geometric execution, enabling more autonomous and reliable scene editing. Its success in diverse, challenging scenarios paves the way for broader deployment in virtual environment creation, robotics, and AR/VR applications. The framework’s modular design and recursive search strategy offer a scalable blueprint for future multi-step, multi-modal AI systems.
Technical Contribution
VULCAN's core innovation lies in integrating a multi-agent architecture with tool-based APIs and recursive backtracking for complex scene rearrangement. It introduces a dedicated planner for global task decomposition, an executor leveraging constraint-based optimization for collision-free placement, and a verifier for visual plausibility assessment. The recursive search algorithm efficiently prunes the exponential solution space, enabling reliable multi-step planning. This combination of external tools, collaborative agents, and advanced search guarantees high accuracy and robustness, surpassing existing single-agent or static methods. The system’s design offers a scalable, interpretable, and adaptable framework for multimodal scene understanding and manipulation.
Novelty
This is the first work to combine multi-agent collaboration, tool APIs, and recursive backtracking specifically for iterative 3D object arrangement tasks. Unlike prior approaches limited to single-step or static analysis, VULCAN dynamically decomposes complex instructions into multi-step plans, ensuring physical plausibility and robustness. Its integration of external visual tools with a dedicated constraint solver and multi-agent cooperation represents a significant leap forward, enabling reliable, multi-object scene editing in complex environments. This novel architecture sets a new benchmark for multi-step, multi-modal scene manipulation.
Limitations
- Currently limited to single-view static scenes, restricting performance in multi-view or dynamic environments. Extending to multi-view scenarios requires additional scene understanding modules.
- The recursive search, while effective, incurs high computational costs in large-scale scenes with many objects, limiting real-time applicability.
- Dependence on predefined APIs and constraints may lead to failures in highly unstructured or novel environments, necessitating more adaptive perception and planning modules.
Future Work
Future directions include extending the framework to multi-view and dynamic scenes, integrating learning-based planning for better scalability, and reducing computational overhead. Exploring end-to-end training with reinforcement learning could improve decision-making robustness. Additionally, incorporating real-world robotic feedback and multi-modal perception will enhance applicability in physical environments. The authors also plan to develop more adaptive APIs and constraint models to handle unstructured scenarios, aiming for fully autonomous scene understanding and manipulation.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在帮朋友整理房间。你会先观察房间里的东西，想好怎么摆放才能既整齐又漂亮。然后，你会一步步把玩具、书本放到合适的地方，比如把积木放到箱子里，把书放到书架上。每次放完一个东西，你还会检查一下，确保没有东西掉下来或者挡路。这个过程就像一个聪明的机器人，它能自己规划每一步，把房间整理得井井有条。它会用一些特别的“工具”帮忙，比如测量距离或检测空间，确保每个东西都放得稳妥。它还会不断检查每一步的效果，如果发现哪里不对，就会退回去重新安排。这样，房间就能被整理得既漂亮又安全，就像你用心打理房间一样。
ELI14 Explained like you're 14
想象你在帮朋友整理房间，你会先看看房间里都有什么，然后决定怎么放。比如，把书放到书架上，把玩具放到箱子里。每放完一样东西，你还会检查一下，确保没有东西掉下来或者挡路。这个过程就像你用脑子规划每一步，确保房间变得又整齐又安全。其实，这个机器人也一样，它能自己想好每一步怎么做，还会用一些特别的工具帮忙，比如测距离或检测空间大小。它还会不断检查自己做的事情，发现不对就会退回去重新安排。这样，房间就变得又漂亮又安全，就像你用心打理的房间一样。
Abstract
Despite the remarkable progress of Multimodal Large Language Models (MLLMs) in 2D vision-language tasks, their application to complex 3D scene manipulation remains underexplored. In this paper, we bridge this critical gap by tackling three key challenges in 3D object arrangement task using MLLMs. First, to address the weak visual grounding of MLLMs, which struggle to link programmatic edits with precise 3D outcomes, we introduce an MCP-based API. This shifts the interaction from brittle raw code manipulation to more robust, function-level updates. Second, we augment the MLLM's 3D scene understanding with a suite of specialized visual tools to analyze scene state, gather spatial information, and validate action outcomes. This perceptual feedback loop is critical for closing the gap between language-based updates and precise 3D-aware manipulation. Third, to manage the iterative, error-prone updates, we propose a collaborative multi-agent framework with designated roles for planning, execution, and verification. This decomposition allows the system to robustly handle multi-step instructions and recover from intermediate errors. We demonstrate the effectiveness of our approach on a diverse set of 25 complex object arrangement tasks, where it significantly outperforms existing baselines. Website: vulcan-3d.github.io