Compositional Chain-of-Thought Prompting for Large Multimodal Models

The evolution of multimodal AI has transitioned from early fusion models to large-scale pretraining of vision-language models like CLIP, Flamingo, and BLIP. These models excel at perception tasks but often lack the capacity for structured, relational reasoning, especially in complex scenes. Scene graphs have been used as a formalized representation to encode objects, attributes, and relations, facilitating reasoning in tasks like visual question answering and scene understanding. However, their reliance on costly annotations limits scalability. Recent efforts focus on leveraging large pre-trained models for zero-shot reasoning, but these models tend to treat scenes as unstructured collections of objects, impairing performance on compositional tasks. Addressing this gap, the current work proposes a method to generate scene graphs dynamically during inference, enabling structured reasoning without annotation overhead.

Despite advancements, large multimodal models still underperform in tasks requiring detailed scene understanding and relational reasoning. The primary challenge is their tendency to perceive scenes as a 'bag of objects,' lacking explicit structural representations. Existing solutions rely on supervised scene graph annotations, which are expensive and not scalable. Moreover, fine-tuning models on scene graph data leads to catastrophic forgetting of pretraining objectives, reducing versatility. Therefore, a scalable, annotation-free approach to enhance compositional reasoning remains an open problem, crucial for applications demanding nuanced scene comprehension such as autonomous navigation, robotics, and complex visual question answering.

The key innovations include: 1) A zero-shot, prompt-based scene graph generation mechanism (Sin) that constructs structured scene representations in JSON format, guiding the model to reason about objects, attributes, and relations. 2) Integrating the generated scene graph as an intermediate reasoning step, which improves the model’s understanding of scene composition without requiring additional training. 3) Demonstrating broad applicability across multiple models (GPT-4V, LLaVA, InstructBLIP) and tasks, with performance gains on diverse benchmarks. This approach uniquely combines the strengths of chain-of-thought prompting with dynamic scene understanding, avoiding costly annotations and fine-tuning, thus offering a scalable solution for structured multimodal reasoning.

• �� Input: image I and task prompt Pin.
• �� Design scene graph prompt Sin, instructing the model to generate a structured JSON scene graph capturing objects, attributes, and relations.
• �� Generate scene graph Sg = f(vϕ(I), l(Pin + Sin)), where vϕ is vision encoder, l is language embedding.
• �� Construct a combined prompt P(2)in including image, scene graph, task prompt, context sentence C, and answer prompt E.
• �� Feed P(2)in into the model to produce response R, which incorporates scene structure understanding.
• �� This process is zero-shot, leveraging the model’s inherent reasoning without fine-tuning, and uses the generated scene graph as an explicit reasoning aid.

Experiments involved applying CCoT to models like GPT-4V, LLaVA-1.5, and InstructBLIP, evaluated on benchmarks including Winoground, WHOOPS!, and SEEDBench. The evaluation metrics included accuracy on compositional tasks, group matching, and relation understanding. Baselines included no prompting, standard chain-of-thought prompting, and models fine-tuned on scene graphs. Ablation studies examined the impact of JSON formatting, scene graph quality, and input length. Results showed consistent performance improvements, with over 10% gains in key metrics, validating the effectiveness of the scene graph-based prompting strategy across models and tasks.

CCoT outperformed baseline prompts across all benchmarks: Winoground group accuracy increased from 58.2% to 69.7%, overall scores improved by 13.5%. InstructBLIP-13B with CCoT achieved a 14.6% performance boost. Ablation confirmed JSON formatting and scene graph quality as critical factors. The method demonstrated robustness across different input lengths (e.g., 128, 512, 1024 tokens) and model sizes, maintaining performance stability. Performance gains were especially notable in tasks involving object relations and scene structure understanding, confirming the hypothesis that structured prompts enhance compositional reasoning.

CCoT can be immediately applied to improve visual question answering, scene understanding, and robotic perception systems. Its zero-shot nature makes it suitable for deployment in resource-constrained environments, reducing reliance on costly annotations. The approach can enhance autonomous navigation, surveillance, and assistive robots by providing structured scene comprehension. Long-term, integrating CCoT with continual learning and multi-task training could lead to more autonomous, reasoning-capable AI systems capable of understanding complex, dynamic environments, advancing fields like autonomous driving, smart surveillance, and human-robot interaction.

The quality of generated scene graphs heavily depends on the underlying model's reasoning ability; complex or ambiguous scenes may produce inaccurate graphs, limiting downstream response quality. In scenes with occlusion or clutter, scene graphs may lack completeness, affecting reasoning depth. The approach's computational overhead for scene graph generation may hinder real-time applications, especially in high-resolution or video data. Additionally, current experiments focus on static images; extending to videos or 3D scenes requires further development. Future work should address these issues to improve robustness and efficiency.