Vision Harnessing Agent for Open Ad-hoc Segmentation
VASA surpasses SAM3 Agent by 14-25% in open ad-hoc segmentation.
Key Findings
Methodology
VASA is a training-free vision-guided ad-hoc segmentation agent that integrates a vision-language model (VLM), a segmentation foundation model, and a visually grounded workflow. Its core mechanism involves reasoning, constructing, and validating solutions through a persistent working mask, planning visual operations, invoking segmentation tools, inspecting results, editing the mask, and recovering from errors.
Key Results
- On the PARS benchmark, VASA improves gIoU and cIoU by 13.5%-25.3% over SAM3 Agent, with xIoU reduced by 25.3%, indicating fewer confusions with semantically related concepts.
- On the RefCOCOm benchmark, VASA outperforms SAM3 Agent by 8.8% in gIoU for part-level concepts and 5.9% for object-level concepts.
- VASA achieves performance comparable to VLPart, a fully supervised method trained on PartImageNet, without task-specific training.
Significance
VASA demonstrates the effectiveness of agentic visual construction in open ad-hoc segmentation, surpassing existing open-vocabulary, reasoning-based, and agentic baselines. This work points to a path for AI agents beyond wrapping foundation models as tools, by programming them with task knowledge, VLM behavior, visual routines, working memory, and failure-aware workflows.
Technical Contribution
VASA achieves long-horizon reasoning through full vision harness engineering, maintaining persistent visual states and offering new engineering possibilities compared to existing methods. Unlike SAM3 Agent, VASA does not rely on prompt refinement but incrementally constructs solutions through visual operations.
Novelty
VASA is the first vision harnessing agent for open ad-hoc segmentation, combining VLM and segmentation foundation models to perform visual construction through a persistent working mask rather than relying solely on prompt refinement.
Limitations
- VASA may require longer reasoning time and more computational resources when handling very complex visual concepts.
- In some cases, VASA may not completely avoid confusion between semantically related concepts.
Future Work
Future research directions include optimizing VASA's reasoning efficiency, reducing computational overhead, and extending its application potential to other visual tasks.
AI Executive Summary
Open ad-hoc segmentation is a significant challenge in computer vision, requiring segmentation without predefined concepts. Existing methods, like SAM3 Agent, primarily rely on prompt refinement but show limitations when handling complex visual concepts.
VASA (Vision-guided Ad-hoc Segmentation Agent) offers a training-free solution by integrating vision-language models and segmentation foundation models. Its core innovation lies in using a persistent working mask for reasoning and construction, allowing complex visual concepts to be incrementally built from simpler visual primitives.
Experimental results show that VASA significantly outperforms existing methods on the PARS and RefCOCOm benchmarks, demonstrating the effectiveness of visual construction in open ad-hoc segmentation. This work provides a new path for AI agents, moving beyond the traditional approach of using foundation models merely as tools.
Deep Analysis
Background
Segmentation technology has made significant progress in computer vision, especially for known concepts where modern foundation models can effectively segment familiar visual wholes. However, segmentation remains challenging for open ad-hoc concepts, which need to be constructed from image evidence during reasoning.
Core Problem
The core problem of open ad-hoc segmentation is segmenting without predefined concepts. In this scenario, users may request arbitrary concepts involving parts, relations, exclusions, or collections, requiring the segmenter to discover relevant concepts at the pixel level and construct solutions.
Innovation
VASA's core innovation lies in its training-free framework, integrating vision-language models and segmentation foundation models to perform reasoning and construction through a persistent working mask. Unlike existing methods, VASA does not rely on prompt refinement but incrementally constructs solutions through visual operations.
Methodology
- �� VASA integrates VLM and segmentation foundation models to perform reasoning and construction through a persistent working mask.
- �� Plans visual operations, invokes segmentation tools, inspects results, edits the mask, and recovers from errors.
- �� Constructs the PARS benchmark, turning part-level labels in PartImageNet into open ad-hoc concepts.
Experiments
The experimental design includes evaluating VASA's performance on the PARS and RefCOCOm benchmarks. The PARS benchmark transforms part-level labels in PartImageNet into open ad-hoc concepts through long-form definition queries. The RefCOCOm benchmark assesses multi-granularity referring segmentation performance.
Results
On the PARS benchmark, VASA improves gIoU and cIoU by 13.5%-25.3% over SAM3 Agent, with xIoU reduced by 25.3%. On the RefCOCOm benchmark, VASA outperforms SAM3 Agent by 8.8% in gIoU for part-level concepts.
Applications
VASA can be applied in scenarios requiring open ad-hoc segmentation, such as dynamic scene understanding in autonomous driving and complex task execution in robotic vision.
Limitations & Outlook
VASA may require longer reasoning time and more computational resources when handling very complex visual concepts. In some cases, VASA may not completely avoid confusion between semantically related concepts.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen, and VASA is like a smart assistant. You tell it you want a specific dish, like fried rice without carrots. VASA doesn't look for a ready-made recipe but selects and excludes ingredients based on your request until it makes the dish you want. This process is like VASA finding the parts you want in an image, excluding the unwanted parts, until it gets the correct segmentation result.
ELI14 Explained like you're 14
Imagine you're playing a game where the task is to find specific items on a map. VASA is like a super-smart assistant; you tell it what to find, and it marks what you want on the map step by step, excluding unrelated parts. Just like in a game, you might want to find a treasure chest without monsters nearby, and VASA helps you find that chest and ensures no monsters are around!
Glossary
Open Ad-hoc Segmentation
The process of segmenting images without predefined concepts.
VASA is used to solve open ad-hoc segmentation problems.
Vision-Language Model
A model that integrates visual and language information for reasoning.
VASA integrates VLM for visual reasoning.
Persistent Working Mask
An image mask that is continuously updated during reasoning.
VASA uses a persistent working mask for visual construction.
PARS Benchmark
A new benchmark for evaluating open ad-hoc segmentation performance.
VASA is evaluated on the PARS benchmark.
RefCOCOm Benchmark
A benchmark for evaluating multi-granularity referring segmentation performance.
VASA is tested on the RefCOCOm benchmark.
Open Questions Unanswered questions from this research
- 1 How to improve VASA's reasoning efficiency without increasing computational overhead?
- 2 How can VASA maintain efficiency when handling more complex visual concepts?
Applications
Immediate Applications
Autonomous Driving
VASA can be used to identify complex objects in dynamic scenes, enhancing safety and efficiency in autonomous driving.
Long-term Vision
Robotic Vision
VASA can help robots execute tasks in complex environments, such as identifying and handling specific items in a factory.
Abstract
Segmentation has become easy when the concept is known, requiring retrieval of a learned visual grounding from text. It remains hard for open ad-hoc concepts, where the grounding may not exist as one learned mask and must often be constructed from image evidence through parts, relations, exclusions, and collections. We propose a Vision-guided Ad-hoc Segmentation Agent (VASA), the first vision harnessing agent for open ad-hoc segmentation. VASA is training-free and couples a VLM agent, a segmentation foundation model, and a visually grounded workflow. Rather than revising text prompts alone, VASA uses a persistent working mask to reason, construct, and validate a solution. It plans visual operations, invokes segmentation tools, inspects results, edits the mask, and recovers from errors. We construct PARS, a new benchmark that turns part-level labels in PartImageNet into open ad-hoc concepts through long-form definition queries. On PARS, VASA outperforms open-vocabulary, reasoning-based, and agentic baselines, surpassing SAM3 Agent by 14-25%. On RefCOCOm, a standard multi-granularity referring segmentation benchmark, VASA improves over SAM3 Agent by 5-9% and over other agentic baselines by up to 20%. These results validate agentic visual construction for open ad-hoc segmentation. Our work points to a path for AI agents beyond wrapping foundation models as tools: Programming them with task knowledge, VLM behavior, visual routines, working memory, and failure-aware workflows.