Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
Trace environment uses scene grammar and executable task programs to enhance multi-domain visual reasoning, improving transferability by 3.5-4% on external benchmarks.
Key Findings
Methodology
This paper introduces Trace, a structured environment built on scene grammars and executable task programs, which decouple visual scene construction from reasoning computation. It encompasses 277 scene grammars across 11 visual domains, generating 1000 tasks. Tasks are represented as tuples of scene grammar, program, answer schema, and reward contract, enabling precise supervision and instance replay. The environment supports controlled semantic and visual variations. Reinforcement learning models trained on 64,000 instances demonstrate significant performance gains—3.51% at 3B and 4.06% at 7B—on 24 external benchmarks, validating the transferability of procedural training.
Key Results
- Models trained with Trace data outperform baselines on 24 benchmarks, with macro-average improvements of 3.51 percentage points for Qwen2.5-VL-3B and 4.06 points for Qwen2.5-VL-7B. The environment's structured task definitions and scene reuse enable robust generalization across diverse visual reasoning tasks, including charts, puzzles, and geometric problems. Ablation studies confirm that scene grammar reuse and task program standardization are key to performance gains.
- The approach demonstrates strong cross-domain transfer, with models maintaining high accuracy despite visual and semantic variations. The explicit task boundary definitions prevent overcounting and ensure meaningful task distinctions. Results indicate that procedural, program-centered environments can significantly enhance the scalability and robustness of visual reasoning models.
- Further analysis shows that the environment's modular design allows for efficient extension—new scene grammars and tasks can be added without disrupting existing structures. The combination of precise supervision, instance replay, and controlled variation underpins the observed improvements, setting a new benchmark for multi-domain visual reasoning.
Significance
This work addresses the critical challenge of generating diverse, verifiable, and transferable visual reasoning data. By structuring tasks through scene grammars and executable programs, it overcomes the limitations of template-based and dataset-dependent approaches. The significant performance improvements on external benchmarks highlight the potential of procedural environments to facilitate scalable, generalizable AI systems. Such advancements are crucial for deploying AI in real-world scenarios requiring robust multi-domain understanding, such as robotics, autonomous vehicles, and intelligent assistants. The environment’s design also paves the way for future research into automated scene and task generation, further expanding the scope of visual reasoning capabilities.
Technical Contribution
The paper introduces a novel task taxonomy that separates scene construction from reasoning logic, enabling stable task units across multiple domains. The environment employs a generator–renderer–verifier pipeline, ensuring exact supervision and instance replay. Canonical task programs are standardized across scene grammars, facilitating multi-task learning and transfer. The approach allows for controlled variation at semantic and visual levels, supporting robust training and evaluation. These innovations collectively advance the state-of-the-art in structured, multi-domain visual reasoning environments, providing both theoretical insights and practical tools for scalable AI development.
Novelty
This is the first work to integrate scene grammar-based scene generation with executable task programs across multiple visual domains, creating a unified, scalable environment for visual reasoning. Unlike prior datasets limited to templates or single scenes, Trace’s modular design supports extensive variation and transferability. The explicit task boundary definitions and program standardization enable precise control over task generation and evaluation, setting a new paradigm for structured, multi-domain reasoning environments. It bridges the gap between synthetic scene graph-based datasets and real-world visual understanding, offering a flexible framework for future AI research.
Limitations
- The environment relies on manually crafted scene grammars and task programs, which may limit scalability to more complex or real-world scenarios. Handling highly ambiguous or cluttered scenes remains challenging, as current validation mechanisms may not catch all semantic ambiguities. The training process is computationally intensive, requiring significant resources for environment generation and model training. Future work should focus on automating scene and task generation, improving validation robustness, and reducing computational costs.
Future Work
Future directions include automating scene grammar and task program learning via neural approaches, expanding the environment to incorporate more complex and realistic scenes, and integrating multimodal data such as language and tactile inputs. Developing more efficient generation and validation pipelines will be crucial for scaling up. Additionally, exploring unsupervised or semi-supervised methods for scene and task discovery could further enhance environment diversity. The ultimate goal is to build AI systems capable of generalizing reasoning skills across an even broader set of real-world scenarios.
AI Executive Summary
Visual reasoning remains a core challenge in artificial intelligence, especially when aiming for models that can generalize across diverse domains. Traditional datasets, often based on templates or limited scene collections, restrict the scope of training and hinder transferability. To address this, the paper introduces Trace, a structured environment that leverages scene grammars and executable task programs to generate a wide variety of verifiable, multi-domain visual reasoning tasks. This environment decouples scene construction from reasoning logic, enabling stable task definitions and instance replay, which are essential for precise supervision and robust training.
Trace encompasses 277 scene grammars across 11 visual domains, producing 1000 tasks with controlled semantic and visual variations. Each task is represented by a shared semantic state that determines the rendered image, prompt, answer, and verifier state, ensuring consistency and enabling exact reward verification. The environment's modular design allows for the reuse of scene components and the addition of new tasks without redefining visual scenes, significantly improving scalability.
Experimental results demonstrate that models trained on 64,000 Trace instances outperform baseline models on 24 external benchmarks, with improvements of 3.51% at 3B and 4.06% at 7B scales. These findings confirm that procedural, program-based training enhances the transferability of visual reasoning models beyond the generated task distribution. The approach offers a promising pathway toward scalable, generalizable AI capable of complex multi-domain understanding.
Looking ahead, future work will focus on automating scene and task generation, expanding to more realistic scenarios, and integrating multimodal inputs. The Trace environment sets a new standard for structured, verifiable, and scalable visual reasoning research, with broad implications for AI applications in robotics, autonomous systems, and intelligent assistants.
Deep Dive
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved language-model reasoning, yet its extension to vision-language models remains constrained by the lack of training data that are simultaneously broad, exactly verifiable, and reproducible. We introduce Trace, a taxonomy-guided environment for multidomain visual reasoning. Trace factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation. A shared semantic state determines the rendered image, prompt, typed answer, verifier state, and replayable instance trace. The resulting environment comprises 1,000 tasks over 277 scene grammars and 11 visual domains, with controlled semantic and visual variation. RLVR on 64,000 Trace instances improves the macro-average across 24 external benchmarks by 3.51 percentage points for Qwen2.5-VL-3B and 4.06 points for Qwen2.5-VL-7B, providing evidence that broad procedural training can transfer beyond the generated task distributions. Project page: https://maveryn.github.io/trace/.