PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

The evolution of vision-language models (VLMs) such as CLIP, GPT-4V, and Gemini has revolutionized multi-modal understanding, enabling tasks like visual question answering, image captioning, and reasoning. Early works focused on high-level semantic tasks, but recent advances have shown their potential for zero-shot generalization across diverse domains. Despite these breakthroughs, applying VLMs to embodied AI—specifically robotic control and spatial reasoning—remains challenging due to their primary text output modality and difficulty in producing precise continuous outputs. Prior approaches often relied on fine-tuning or separate perception modules, which limited scalability and adaptability. This paper addresses these gaps by proposing a novel visual prompting framework that leverages the inherent decision-making ability of VLMs, aiming for zero-shot control in complex spatial tasks.

The core challenge lies in enabling VLMs to output continuous spatial data—such as coordinates, trajectories, or actions—without task-specific fine-tuning. Existing models excel at selecting discrete categories but struggle with precise, grounded continuous outputs essential for robotic manipulation and navigation. Additionally, the difficulty in representing spatial concepts visually and the high computational cost of iterative refinement pose significant hurdles. Overcoming these limitations requires a method that can effectively translate spatial tasks into a form compatible with pre-trained models, leveraging their decision-making capacity while maintaining efficiency and generalization. Addressing this problem is crucial for deploying autonomous systems capable of versatile, real-world operations without extensive retraining.

The paper's key innovations include: 1) transforming spatial control tasks into visual question answering by annotating images with candidate proposals, 2) employing an iterative refinement process that samples, projects, and evaluates proposals across multiple rounds, and 3) fitting proposal distributions dynamically to guide the search towards optimal solutions without fine-tuning. This approach differs from prior methods by directly leveraging the pre-trained VLM's decision-making in a multi-round, visual-guided manner, enabling zero-shot spatial reasoning. The integration of proposal sampling, visual annotation, and distribution fitting creates a flexible framework adaptable to diverse tasks, marking a significant step forward in embodied AI.

• �� Define tasks as a visual question answering problem, inputting images and natural language instructions.
• �� Generate candidate proposals (actions/positions) from a probability distribution, initially broad.
• �� Map proposals onto the image via projection, drawing visual markers and labeling them with textual identifiers.
• �� Input the annotated image and task description into the VLM, obtaining relevance scores or selections.
• �� Fit a new proposal distribution based on the model's outputs, focusing on promising regions.
• �� Repeat the sampling, annotation, and evaluation process iteratively, refining the proposals.
• �� Use parallel calls to improve robustness, aggregating multiple candidate solutions.
• �� Continue until convergence or maximum iterations, then execute the best action.
• �� Validate in real robotic platforms and simulated environments, measuring success rates and efficiency.

The experiments involve robotic navigation and manipulation tasks on mobile manipulators, Franka arms, and simulated domains like RAVENS. Metrics include success rate, steps to completion, and grasp success. The evaluation compares single-pass, multi-iteration, and parallel strategies, demonstrating that iterative refinement and parallel calls significantly enhance performance. Additional tests on spatial localization (RefCOCO) assess the model's reasoning accuracy. Ablation studies analyze the impact of proposal quantity, prompt design, and iteration count, confirming the robustness and scalability of the approach across diverse settings.

PIVOT achieves up to 75% success in navigation tasks, outperforming baseline methods by 50%. In manipulation, success rates for grasping reach 67%, with average steps reduced from 4.5 to 3.0. Spatial localization accuracy exceeds 85% on RefCOCO, with minimal iteration. The iterative process consistently improves performance, and parallel calls mitigate local minima, demonstrating robustness. The results validate that large VLMs can perform complex spatial reasoning and control without fine-tuning, highlighting the method's generalizability.

This approach enables zero-shot robotic control, spatial reasoning, and visual grounding in real-world applications such as autonomous navigation, industrial automation, and assistive robotics. It requires only pre-trained VLMs and standard visual inputs, making it adaptable to various platforms. Long-term, integrating reinforcement learning and multi-modal data could further enhance autonomy, enabling robots to learn from interactions and adapt to dynamic environments, ultimately transforming autonomous systems across industries.

Current limitations include reliance on proposal quality, sensitivity to initial distributions, and high computational costs due to multiple iterations. Performance drops in cluttered or occluded scenes, and real-time application remains challenging. Future work should focus on improving proposal sampling, reducing inference overhead, and enhancing robustness in dynamic, complex environments. Addressing these issues is essential for deploying PIVOT in practical, real-time systems.