Causally Steered Diffusion for Automated Video Counterfactual Generation
Proposes CSVC, a causal prompt optimization framework guiding diffusion models to generate causally consistent counterfactual videos, improving effectiveness and quality.
Key Findings
Methodology
The approach encodes predefined causal graphs into natural language prompts, which are optimized via a textual gradient-based method (TextGrad). These prompts steer latent diffusion models (LDMs) during inference, ensuring generated videos adhere to causal structures. The process involves: 1) constructing causal graphs and generating initial prompts with GPT-4; 2) applying a VLM-based textual loss to evaluate causal alignment; 3) iteratively refining prompts through TextGrad, guided by VLM feedback; 4) evaluating causal effectiveness and minimality metrics. This framework operates as a black-box, requiring no internal model access or fine-tuning, making it broadly applicable across different diffusion-based video editing systems.
Key Results
- On CelebV-Text, CSVC achieved an average causal effectiveness of 78.4%, outperforming baseline methods (~55%). It improved interventions like age and gender by about 20%, with LPIPS dropping to 0.179, indicating reduced visual discrepancies. Across multiple interventions, causal consistency exceeded 80%, while video quality metrics (FVD, DOVER) remained comparable or better than baselines. Ablation studies confirmed that causal decoupling prompts boosted effectiveness by 5-10% without model fine-tuning.
- The method maintained high temporal coherence and visual fidelity, with minimal artifacts. It demonstrated robustness across different datasets and intervention types, including beard and baldness. The prompt optimization process converged within a few iterations, showing efficiency. These results validate the framework's ability to generate realistic, causally faithful counterfactual videos in real-world scenarios.
- Evaluation metrics based on VLM feedback provided interpretable, scalable measures of causal alignment and minimality, complementing traditional quality scores. The approach's black-box nature ensures easy integration with existing systems, enabling practical deployment in digital media, healthcare, and beyond.
Significance
This work advances the integration of causal reasoning into generative video models, addressing a key challenge in producing trustworthy, controllable counterfactual scenarios. By encoding causal priors into prompts and optimizing them via VLM feedback, CSVC enables high-fidelity, causally consistent modifications without retraining or internal model access. Its black-box design broadens applicability, making it suitable for diverse real-world applications such as digital media creation, medical simulation, and AI-driven storytelling. The framework also introduces new evaluation metrics for causal effectiveness and minimality, fostering more rigorous assessment of counterfactual fidelity. Overall, this approach paves the way for more explainable and reliable generative AI systems that respect underlying causal structures.
Technical Contribution
The core innovation lies in encoding causal graphs into natural language prompts, which are then optimized via textual gradients guided by a VLM. This enables explicit control over causal relationships during inference, bypassing the need for model fine-tuning. The introduction of a differentiable textual loss for causal alignment, combined with a prompt optimization strategy (TextGrad), allows high-dimensional latent spaces to be steered towards causally meaningful outputs. The framework's black-box compatibility and the development of quantitative metrics for causal effectiveness and minimality further distinguish this work from prior art, offering a scalable, interpretable solution for causal video editing.
Novelty
This is the first framework to embed predefined causal graphs directly into text prompts for guiding diffusion-based video editing. Unlike existing methods that rely on fine-tuning or auxiliary modules, CSVC leverages prompt optimization via textual feedback, enabling explicit causal control in a black-box setting. The integration of causal decoupling prompts and the development of new evaluation metrics for causal fidelity represent significant innovations, setting a new standard for controllable, trustworthy counterfactual video generation.
Limitations
- The reliance on predefined causal graphs limits flexibility; complex or dynamic causal structures are challenging to encode accurately. The method may struggle with multi-layered or uncertain causal relationships, reducing applicability in highly complex scenarios.
- Computationally, the iterative prompt optimization process via VLM feedback is resource-intensive, hindering real-time applications. Further efficiency improvements are needed for deployment in interactive systems.
- The approach assumes accurate causal graphs and effective prompt generation; errors in these components can lead to incorrect or unrealistic counterfactuals. Robustness to such errors remains an open challenge.
Future Work
Future directions include automating causal graph learning from data, extending the framework to handle dynamic causal relationships, and reducing computational overhead. Integrating multi-modal cues (audio, text) for richer causal modeling and exploring real-time applications are also promising avenues. Additionally, developing more sophisticated evaluation metrics and user-controllable interfaces will enhance practical usability and trustworthiness.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里做菜，每个步骤都遵循一定的规则，比如先切菜，再炒菜。现在，如果你想让菜变得更甜，你可以调整一些步骤，比如加糖，而保持其他步骤不变。这就像给视频中的人物“变年轻”或“变男”，你只需要告诉厨师（模型）一些新规则，它就会按照新的指令做出不同的菜肴。我们用一种聪明的方法，先理解这些规则，然后不断调整指令，让厨师做出符合新规则的菜。这种方法让厨房变得更智能，也能做出更多不同的菜肴，满足不同的需求。
ELI14 Explained like you're 14
你知道在学校里，老师会告诉你一些做事的规则，比如数学题要按步骤做。现在，想象你有一台神奇的相机，可以拍出不同的照片，只要你告诉它一些“秘密规则”。比如，你说：“让这个女孩变得更年轻”，它会根据你说的“秘密规则”调整照片。但如果你还想让她变得更帅或者更有胡子，你得告诉相机新的规则。我们用一种聪明的办法，不断调整这些“秘密规则”，让相机拍出符合你想象的照片。这样，你就可以用它做出各种不同的照片，既符合你的想法，又看起来很自然。
Abstract
Adapting text-to-image (T2I) latent diffusion models (LDMs) to video editing has shown strong visual fidelity and controllability, but challenges remain in maintaining causal relationships inherent to the video data generating process. Edits affecting causally dependent attributes often generate unrealistic or misleading outcomes if these relationships are ignored. In this work, we introduce a causally faithful framework for counterfactual video generation, formulated as an Out-of-Distribution (OOD) prediction problem. We embed prior causal knowledge by encoding the relationships specified in a causal graph into text prompts and guide the generation process by optimizing these prompts using a vision-language model (VLM)-based textual loss. This loss encourages the latent space of the LDMs to capture OOD variations in the form of counterfactuals, effectively steering generation toward causally meaningful alternatives. The proposed framework, dubbed CSVC, is agnostic to the underlying video editing system and does not require access to its internal mechanisms or fine-tuning. We evaluate our approach using standard video quality metrics and counterfactual-specific criteria, such as causal effectiveness and minimality. Experimental results show that CSVC generates causally faithful video counterfactuals within the LDM distribution via prompt-based causal steering, achieving state-of-the-art causal effectiveness without compromising temporal consistency or visual quality on real-world facial videos. Due to its compatibility with any black-box video editing system, our framework has significant potential to generate realistic 'what if' hypothetical video scenarios in diverse areas such as digital media and healthcare.