ViperGPT: Visual Inference via Python Execution for Reasoning
ViperGPT uses code generation with GPT-3 Codex to perform visual reasoning without training, achieving state-of-the-art zero-shot results.
Key Findings
Methodology
ViperGPT integrates large language models (like GPT-3 Codex) via API calls to pre-trained vision and knowledge modules, generating Python code that executes complex reasoning tasks. It transforms visual queries into programs, leveraging Python's control flow, mathematical operations, and external modules (e.g., GLIP, MiDaS, X-VLM). The approach is training-free, enabling multi-task generalization and interpretability. The system inputs images/videos and textual queries, producing diverse outputs such as text, regions, or choices, with explicit step-by-step reasoning demonstrated through generated code.
Key Results
- On RefCOCO and RefCOCO+ datasets, ViperGPT achieves 72.0% and 67.0% accuracy respectively, surpassing end-to-end models. On GQA, it reaches 48.1% accuracy in zero-shot mode. Combining external knowledge via GPT-3 boosts OK-VQA performance significantly. Ablation studies confirm that code flexibility and API richness are critical for performance. Across multiple tasks, ViperGPT demonstrates strong generalization and interpretability, outperforming traditional models in complex reasoning scenarios.
- The experiments validate that program synthesis enables transparent reasoning pathways, allowing step-by-step inspection. The approach maintains high accuracy without task-specific training, highlighting the potential of modular, code-based AI for diverse visual tasks. Results show that leveraging language models for program generation is a promising direction for scalable, interpretable AI.
- Ablation results reveal that the expressiveness of generated code and API module diversity are key to success. The system's ability to handle multi-modal inputs and external knowledge further enhances its robustness. Overall, ViperGPT sets a new benchmark for zero-shot visual reasoning, combining flexibility, interpretability, and high performance.
Significance
This work addresses core limitations of end-to-end neural models—lack of interpretability, poor generalization, and training costs—by proposing a modular, code-driven framework. It leverages the strengths of large language models to synthesize explicit reasoning programs, enabling transparent multi-step inference. The approach bridges the gap between neural perception and symbolic reasoning, opening avenues for scalable, explainable AI in complex visual domains. Its ability to perform multiple tasks without retraining marks a significant step toward versatile autonomous systems, with broad implications for robotics, autonomous vehicles, and intelligent surveillance.
Technical Contribution
ViperGPT's main innovation lies in transforming visual reasoning into Python program synthesis driven by GPT-3 Codex. It employs an API-based modular design, allowing dynamic composition of perception and knowledge modules. The approach eliminates the need for end-to-end training, instead relying on language models' prior knowledge for program generation. It supports logical operations, control flow, and mathematical computations, enabling complex multi-step reasoning. This framework enhances transparency, reusability, and scalability, providing a new paradigm for multi-modal AI development.
Novelty
This study is the first to utilize large language models directly for generating executable visual reasoning programs, integrating API calls for flexible module composition. Unlike prior neural module networks requiring supervised training, ViperGPT leverages pre-trained models and code synthesis to achieve zero-shot multi-task performance. Its emphasis on explicit, interpretable programs distinguishes it from opaque end-to-end models, representing a significant leap in modular, explainable AI.
Limitations
- Dependence on the quality of pre-trained modules; poor module performance limits overall accuracy.
- In highly ambiguous or novel scenarios, generated programs may fail to cover all reasoning paths, reducing reliability.
- Computational overhead of large models and program execution may hinder real-time deployment.
Future Work
Future directions include expanding API modules for broader tasks, improving program robustness and efficiency, integrating reinforcement learning for adaptive program synthesis, and developing richer multi-modal datasets. Enhancing model scalability and deploying in real-world applications like autonomous systems will be key goals.
AI Executive Summary
ViperGPT introduces a transformative approach to visual reasoning by harnessing large language models (like GPT-3 Codex) to generate explicit Python programs that perform complex inference tasks. Unlike traditional end-to-end neural networks, this framework decomposes visual queries into step-by-step logical operations, making the reasoning process transparent and interpretable. It leverages an API of pre-trained modules—such as object detectors, depth estimators, and external knowledge sources—allowing dynamic composition tailored to each query.
The core innovation lies in transforming visual understanding into code synthesis. When a query is posed, ViperGPT prompts the language model to produce a Python function that calls relevant modules, performs logical operations, and computes the answer. This code is then executed within a standard Python interpreter, ensuring faithful and explainable reasoning. The approach benefits from the prior training of language models on internet-scale code, enabling zero-shot generalization across diverse tasks.
Experimental results demonstrate that ViperGPT achieves state-of-the-art zero-shot performance on multiple benchmarks. For example, on the RefCOCO dataset, it surpasses previous models with 72.0% accuracy, and on GQA, it reaches 48.1%. When integrating external knowledge, it significantly improves answers in knowledge-dependent tasks like OK-VQA. These results highlight the method’s robustness, flexibility, and interpretability, making it a promising direction for scalable AI systems.
The framework's transparency allows users to inspect generated programs, fostering trust and facilitating debugging. Its modular design supports easy extension with new perception or reasoning modules, promoting rapid adaptation to evolving tasks. While computational costs and reliance on module quality remain challenges, ongoing research aims to optimize efficiency and expand capabilities. Overall, ViperGPT paves a new path toward explainable, multi-task visual AI, with broad implications for robotics, autonomous vehicles, and intelligent surveillance.
Deep Analysis
Background
近年来，视觉理解技术不断演进，深度学习模型如Transformers在目标检测、图像识别等任务中取得显著突破。端到端模型如ViT和Transformer-based检测器在性能上表现优异，但缺乏明确的推理路径，难以解释其决策过程。神经模块网络（Neural Module Networks）曾试图通过显式模块化实现可解释推理，但训练复杂、扩展困难。大规模预训练模型（如GPT-3、CLIP）推动多模态理解，但多任务融合仍依赖端到端训练，限制了模型的可解释性和灵活性。ViperGPT结合程序合成与API调用，提出无需训练的模块化推理框架，旨在解决复杂视觉推理中的可解释性和泛化问题，为未来多模态AI提供新思路。
Core Problem
传统视觉推理模型多为端到端架构，缺乏推理步骤的透明性，难以解释复杂推理过程，且在多任务、多模态场景中泛化不足。训练成本高，模型难以扩展到新任务或场景。尤其是在需要结合外部知识或进行多步推理时，端到端模型表现有限。如何设计一个既具备良好解释性，又能灵活应对多样任务的系统，成为核心难题。解决方案需实现任务拆解、模块调用和逻辑推理的明确表达，同时保证系统的可扩展性和效率。
Innovation
ViperGPT的创新点包括：1）利用GPT-3 Codex自动生成Python程序，将复杂推理转化为代码执行，增强可解释性；2）通过API调用预训练视觉和知识模块，实现多模态信息的动态组合，避免端到端训练；3）引入Python控制流和内置函数，支持逻辑推理和数学运算，适应多任务场景；4）无需训练，直接零-shot实现多任务性能最优。这些创新突破了传统神经网络的限制，提供了更灵活、透明的推理框架。
Methodology
- �� 输入：图像或视频，文本查询。
- �� 任务：利用大规模语言模型（如GPT-3 Codex）根据API规范生成Python程序。
- �� API设计：定义视觉模块（find、simple_query、verify_property）和知识模块（llm_query），支持多模态任务。
- �� 生成：模型根据查询和API提示，自动合成程序代码。
- �� 执行：用Python解释器运行代码，调用预训练模型实现视觉和知识推理。
- �� 输出：多样化（文本、区域、选择题），支持复杂推理和多步任务。
- �� 优势：程序透明、逻辑清晰、易于调试和扩展。
Experiments
采用RefCOCO、RefCOCO+、GQA和OK-VQA等公开数据集，评估视觉定位、问答和外部知识融合能力。比较端到端模型和ViperGPT的零-shot性能，分析不同模块对性能的影响。设置合理超参数（API调用频次、模型温度），进行消融实验验证程序生成的关键作用。通过多任务测试，验证模型在复杂推理场景中的泛化能力和可解释性。
Results
在RefCOCO和RefCOCO+数据集上，ViperGPT分别达成72.0%和67.0%的准确率，显著优于端到端模型。GQA任务中，零-shot准确率达48.1%，优于其他无训练模型。结合外部知识（如GPT-3）后，OK-VQA的回答正确率大幅提升。 Ablation研究显示，程序的表达能力和API丰富性是性能提升的关键。模型在多模态、多任务环境中表现出强大泛化能力，验证了模块化推理的有效性。
Applications
该方法适用于自动驾驶中的场景理解、机器人视觉导航、智能监控等领域。只需提供API和预训练模块，无需训练即可快速部署，适应多变环境。未来可结合强化学习优化程序生成策略，提升推理效率和准确性，推动智能系统的自主决策能力。
Limitations & Outlook
依赖预训练模型性能，模块不完善会影响整体效果。复杂场景中程序可能无法覆盖所有推理路径，导致误差。高计算成本限制了实时应用，需优化推理速度。未来需增强模型鲁棒性和效率，扩展API接口，提升实用性。
Plain Language Accessible to non-experts
想象你在厨房做一道复杂的菜肴。你会先查食谱，找到所有需要的食材和步骤，然后逐一准备。每个步骤都可以用简单的指令，比如“切菜”、“煮汤”。如果你有一台智能厨房助手，它可以帮你列出所有步骤，甚至帮你查找食材的详细信息。ViperGPT就像这个智能助手，它用一套“食谱”——也就是程序——来一步步完成复杂的视觉任务。它会先找到图片中的物体，确认它们的属性，然后根据指令做出判断或回答问题。这样，整个过程变得清晰、可控，也更容易理解和改进。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。你不能一下子把所有拼图都拼好，而是要一步步找拼图块，把它们拼成完整的图片。每次你找到一块拼图，你会先确认它是什么，然后决定下一步怎么拼。ViperGPT就像这个拼图高手，它会用一套“聪明的机器人”帮忙找到图片里的东西，比如“狗”或者“车”，然后用一段“指令”告诉自己下一步怎么做，比如“找出最大的车”。它不用一次性学会所有东西，而是用一套“指令”逐步完成任务。这样，不仅做得快，还能让你知道每一步是怎么做到的，特别聪明又透明！
Glossary
Large Language Model (大规模语言模型)
一种基于海量文本数据训练的模型，能理解和生成自然语言，支持代码生成和推理。在本文中，用于程序合成和知识查询。
ViperGPT利用GPT-3 Codex作为程序生成的核心工具。
API (应用程序接口)
一组定义不同模块功能的规范，允许程序调用外部预训练模型实现特定任务。在本文中，API封装视觉和知识模块。
API设计使得程序能灵活调用不同视觉模块。
Zero-shot learning (零-shot学习)
模型在未见过特定任务或数据的情况下，直接实现任务目标的能力。
ViperGPT在未专门训练的情况下，达成多任务最优性能。
Neural Module Networks (神经模块网络)
将复杂任务拆解成可重用的神经网络模块，通过组合实现多样化推理。
ViperGPT借鉴其模块化思想，但用代码替代训练。
Program synthesis (程序合成)
自动生成满足特定需求的程序代码的技术。
ViperGPT利用大模型自动生成Python程序。
Open Questions Unanswered questions from this research
- 1 如何进一步提升程序生成的鲁棒性，避免在极端复杂场景下出错。
- 2 API设计的模块是否能涵盖所有实际需求，未来如何扩展。
- 3 模型在实时应用中的效率优化路径。
Applications
Immediate Applications
智能监控
利用ViperGPT实现场景中的目标检测和行为识别，提升监控系统的智能化水平，支持多模态数据分析。
机器人导航
通过视觉推理帮助机器人理解环境，进行路径规划和目标识别，增强自主决策能力。
Long-term Vision
自主智能系统
打造具备复杂推理能力的自主系统，实现无人驾驶、智能制造等高级应用，推动AI普及。
Abstract
Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.