Synthesizing Visual Concepts as Vision-Language Programs

Recent advances in multimodal AI have seen the emergence of large-scale vision-language models (VLMs) like CLIP, ALIGN, and InternVL, which excel at perception and semantic understanding. However, these models lack systematic reasoning capabilities, often producing inconsistent outputs in tasks requiring logical inference, such as rule induction, compositional reasoning, and abstract concept learning. Neuro-symbolic AI offers a promising solution by integrating neural perception with symbolic inference, providing interpretability and robustness. Prior work like Wüst et al. introduced program synthesis for visual concept induction but relied on domain-specific detectors, limiting generalization. The challenge remains to develop a flexible, end-to-end framework that can automatically induce symbolic rules directly from natural images, enabling scalable and explainable reasoning across diverse visual domains.

Despite the success of VLMs, their inability to perform reliable systematic reasoning remains a bottleneck. Existing methods either depend heavily on handcrafted detectors, which are domain-specific and lack scalability, or on extensive chain-of-thought prompting, which is computationally expensive and prone to contradictions. The core problem is how to leverage the perceptual power of VLMs while enabling explicit, interpretable, and systematic reasoning without manual intervention. Addressing this gap requires a framework that can automatically induce symbolic representations from raw images, support complex logical operations, and remain generalizable across tasks and domains, all while maintaining computational efficiency.

This paper introduces VLP, a neuro-symbolic framework that combines VLMs with program synthesis. Key innovations include: 1) automatic symbol grounding from images using pretrained VLMs, eliminating reliance on handcrafted detectors; 2) a flexible DSL that integrates visual perception functions with logical operators, supporting structured reasoning; 3) probabilistic program search via PCFG that guides the induction of symbolic rules based on few labeled examples; 4) a decoupling of perception and reasoning, enabling modularity and interpretability. These innovations collectively enable the induction of human-interpretable rules directly from natural images, significantly advancing the state of neuro-symbolic visual reasoning.

• �� First, use pretrained VLMs (e.g., InternVL-8B, Qwen-2.5B) to analyze each image, extracting symbols such as objects, properties, and actions, forming a structured symbolic pool. • Second, define a DSL that encodes visual functions (get_objects, get_actions) and logical operators (and, or, exists). This language supports flexible program construction. • Third, employ a probabilistic context-free grammar (PCFG) to generate candidate programs, assigning probabilities based on symbol occurrence frequencies in positive and negative examples. • Fourth, perform program search by evaluating candidate programs on the dataset, selecting the highest accuracy and likelihood. • Fifth, execute the best program on new images to infer labels, ensuring logical consistency and interpretability.

The framework was evaluated on synthetic datasets (CLEVR-Hans3) and real-world datasets (COCOLogic, Bongard). Baselines included direct prompting of VLMs and dedicated reasoning models. Metrics focused on balanced accuracy, with support samples ranging from 6 to 20. Ablation studies examined the impact of symbol grounding, DSL design, and program search. Results showed VLP consistently outperformed baselines, with accuracy gains of up to 26% on complex tasks. The experiments demonstrated robustness to noise, generalization across domains, and the ability to detect shortcut learning through program transparency.

VLP achieved an average accuracy increase of 13.5% over direct prompting across datasets, with the highest gains on CLEVR-Hans3 (26.1%). It maintained high performance with limited samples and high logical complexity, outperforming specialized reasoning models. The programmatic approach enabled transparent reasoning, allowing detection of reasoning shortcuts and improving trustworthiness. The experiments confirmed that structured symbolic reasoning significantly enhances robustness and interpretability, especially in out-of-distribution scenarios.

This approach can be applied in automated visual reasoning systems, such as intelligent surveillance, autonomous vehicles, and robotic perception, where explainability and robustness are critical. It also benefits knowledge extraction and question answering systems, providing transparent reasoning paths. The minimal data requirement makes it suitable for low-resource settings, and its generality supports adaptation to diverse domains, including medical imaging and industrial inspection.

The method relies heavily on accurate symbol grounding; noisy or ambiguous images can impair rule induction. The program search process is computationally intensive, especially for complex tasks, limiting scalability. The design of DSL and symbols may require task-specific tuning, reducing out-of-the-box applicability. Handling dynamic scenes and multi-step reasoning remains challenging, necessitating further research to improve efficiency and flexibility.