VCIFBench: Evaluating Complex Instruction Following for Video Understanding

The evolution of multimodal large language models (MLLMs) such as Dai et al. (2023) and Comanici et al. (2025) has significantly advanced the understanding and reasoning capabilities across diverse data modalities, including images and videos. These models leverage large-scale pretraining on multimodal datasets to excel at tasks like visual question answering, captioning, and scene understanding. Despite these achievements, current benchmarks predominantly focus on task-specific performance with simple prompts, providing limited insight into models’ ability to follow complex instructions involving multiple constraints. Prior works like Zhou et al. (2023) and He et al. (2024) have introduced rule-based and multi-task instruction sets, but they lack comprehensive evaluation of multi-constraint satisfaction and conflict detection in video contexts. As real-world applications demand models to adhere to detailed content, format, and stylistic requirements, there is a pressing need for benchmarks that evaluate these nuanced capabilities systematically.

Existing models demonstrate proficiency in isolated tasks but struggle with complex, multi-constraint instructions in video understanding. Challenges include accurately grounding content, respecting format and style constraints, and detecting conflicts among multiple requirements. These limitations hinder deployment in practical scenarios where precise compliance is critical, such as automated content moderation, assistive technologies, and content generation. The core issue is the lack of a systematic, scalable evaluation framework that captures the intricacies of multi-constraint adherence, especially in temporally distributed video data. Consequently, models often produce outputs that satisfy some constraints but violate others, leading to unreliable performance and reduced trustworthiness.

This work introduces several key innovations: 1) VCIFBench, a benchmark integrating diverse datasets and constraints to evaluate multi-dimensional instruction following; 2) a hybrid verification pipeline combining rule-based checks, parameter extraction, and LLM judgment, improving verification reliability; 3) the Conflict-100 dataset, designed to diagnose a model’s ability to detect unsatisfiable instructions and conflicts; 4) leveraging DPO for preference-based fine-tuning, which enhances models’ adherence to complex instructions. These innovations collectively address the limitations of prior benchmarks by enabling detailed, multi-layered assessment of models' reasoning, grounding, and conflict detection capabilities in video understanding.

• �� Data collection: Integrate five datasets (TempCompass, MMWorld, NExT-QA, YouCook2, Video-MME), focusing on short videos (<2 min).• Instruction generation: Use GPT-5.2 to create open-ended task prompts, verified with Gemini-2.5 Pro for content support, manually refined for consistency.• Constraint design: Categorize into content, format, style, and structure, totaling 40 types, including length, lexical diversity, chronological order, and rhetorical style.• Task construction: Combine prompts with constraints using And, Chain, Selection, and Nested patterns, ensuring diverse instruction complexity.• Verification pipeline: Implement rule-based checks for surface constraints, parameter extraction via model prompts, and LLM judgment (GPT-5) for semantic and reasoning constraints.• Evaluation metrics: Measure IPR, CPR, and conflict detection accuracy, analyze failure modes, and assess robustness under option order and input scale variations.• Experimental setup: Test 10 models, tune with DPO, perform ablation studies, and analyze performance across constraints and complexity levels.

The experimental setup involves evaluating 10 models, including proprietary (Gemini-3-Pro, GPT-5.4) and open-source (Qwen3-VL-30B, InternVL3.5-14B). Using a standardized input pipeline, videos are sampled at 0.5 fps, capped at 1080p, with a temperature of 0.1 for generation. The models are tested on the 306-sample satisfiable set, with metrics including IPR, CPR, and failure severity. Ablation studies vary visual input resolution and frame rate, assessing their impact on performance. Preference tuning (DPO) is applied to evaluate improvements. Additional tests include option order swaps and multi-constraint combinations to analyze robustness. Results highlight that even strong models struggle with joint constraint satisfaction, especially under increased complexity, with notable failure modes such as boundary violations and global conflicts.

The results show that the highest IPR among models is approximately 52%, with CPR around 85%. As the number of constraints increases from 2 to 8, IPR drops sharply from 57% to 7.5%. Preference optimization improves adherence by about 10%. Strong models like Gemini-3-Pro and GPT-5.4 tend to make boundary errors, violating explicit constraints, while weaker models often produce incoherent outputs or fail globally. The robustness tests reveal that models like Gemini-3-Pro maintain stability under option order changes, whereas others like Qwen3-VL-8B are more sensitive. Visual input scale variations do not significantly improve joint constraint satisfaction, indicating that input size alone is insufficient. Overall, the experiments underscore the persistent difficulty of multi-constraint adherence and conflict detection in video understanding models.

This benchmark facilitates the development of more reliable multimodal models capable of complex instruction following, applicable in automated content moderation, intelligent virtual assistants, and content creation tools. It helps identify model weaknesses, guiding targeted improvements. Long-term, such advancements could enable AI systems that autonomously generate, verify, and refine multimedia content with high fidelity and compliance, transforming industries like entertainment, education, and digital marketing. Additionally, it supports research into explainable AI and conflict-aware reasoning, crucial for deploying trustworthy AI in safety-critical applications.

Despite its comprehensive design, the benchmark still faces challenges: models often fail to recognize global conflicts, especially in complex, multi-constraint scenarios. The verification process, relying heavily on LLM judgment, may introduce biases or inaccuracies. The datasets, while diverse, are limited to short videos and specific task types, restricting generalization. Computational costs for multi-layer validation are high, and the framework's scalability to longer videos or real-time applications remains to be tested. Future work should address these issues by developing more efficient verification algorithms and expanding dataset diversity.