EditCLEVR: A Paired-Scene Intervention Benchmark for Compositional Faithfulness of Object-Centric Representations
EditCLEVR: A benchmark for evaluating object-centric representations under semantic edits.
Key Findings
Methodology
EditCLEVR evaluates object-centric representations through paired CLEVR-style scenes, assessing behavior under semantic edits. Each pair includes a known attribute change or no-edit re-render for drift measurement, using probe-free diagnostics and probe-decoded semantic faithfulness metrics.
Key Results
- On CoGenT-OOD-core, models with ground-truth masks showed performance degradation, indicating semantic faithfulness isn't solely dependent on mask source.
- Experiments reveal locality or stability alone may overstate semantic faithfulness.
- Baseline evaluations across models show mask source only partially impacts performance.
Significance
EditCLEVR provides a new framework to directly test object-centric representations under semantic edits. It emphasizes compositional generalization and OOD transfer, aiding researchers in understanding representation stability and faithfulness.
Technical Contribution
EditCLEVR introduces Scene-Graph Intervention Accuracy (SGIA) and Delta-SGIA, offering strict object-attribute scene-graph metrics. It also demonstrates compositional OOD degradation under ground-truth masks.
Novelty
EditCLEVR is the first benchmark to directly test object-centric representations under semantic edits. Unlike existing methods, it emphasizes compositional faithfulness and OOD transfer.
Limitations
- EditCLEVR is synthetic, limited to CLEVR-derived scenes, four discrete attributes, and single-object edits.
- Baseline evaluations cover only a subset of object-centric, generative, or VLM systems.
Future Work
Future work could extend to natural images, relational or continuous edits, multi-object edits, and stronger discovery sources.
AI Executive Summary
EditCLEVR is a novel benchmark for evaluating object-centric representations under semantic edits. Existing evaluations often focus on segmentation or single-image factor prediction but do not directly test object representations under semantic edits. EditCLEVR offers a rigorous evaluation framework through paired CLEVR-style scenes.
The benchmark consists of scene pairs, each containing a known attribute change or no-edit re-render. Using probe-free diagnostics and probe-decoded semantic faithfulness metrics, EditCLEVR assesses whether representation changes localize to the edited object, unedited objects remain stable, and the decoded object-attribute graph changes only in the intended way.
EditCLEVR's contributions include introducing Scene-Graph Intervention Accuracy (SGIA) and Delta-SGIA, providing strict object-attribute scene-graph metrics. Baseline evaluations show compositional OOD degradation persists under ground-truth masks. Future work could extend to natural images and multi-object edits.
Deep Analysis
Background
Object-centric learning aims to represent scenes through objects and their attributes, allowing familiar factors to be recombined in new scenes. Existing evaluations typically focus on segmentation, single-image factor prediction, or downstream accuracy but do not directly test object representations under semantic edits.
Core Problem
Existing evaluations do not directly test object representations under semantic edits. Segmentation metrics ask whether objects are discovered, factor probes ask whether attributes are decodable in a single image, and downstream accuracy mixes representation quality with task-specific shortcuts.
Innovation
EditCLEVR offers a rigorous evaluation framework through paired CLEVR-style scenes. Each pair includes a known attribute change or no-edit re-render for drift measurement. Using probe-free diagnostics and probe-decoded semantic faithfulness metrics, EditCLEVR assesses whether representation changes localize to the edited object.
Methodology
- �� Evaluate using paired CLEVR-style scenes
- �� Each pair includes a known attribute change or no-edit re-render
- �� Use probe-free diagnostics to assess whether representation changes localize to the edited object
- �� Use probe-decoded semantic faithfulness metrics to assess whether the decoded object-attribute graph changes only in the intended way
Experiments
Experimental design includes baseline evaluations using ground-truth-mask backbones, learned-slot models, SAM 2 + frozen ViT models, and one mask-feature hybrid. Evaluates object-centric representations under semantic edits.
Results
Experiments reveal locality or stability alone may overstate semantic faithfulness. Baseline evaluations show compositional OOD degradation persists under ground-truth masks.
Applications
EditCLEVR can be used to evaluate object-centric representations under semantic edits, aiding researchers in understanding representation stability and faithfulness.
Limitations & Outlook
EditCLEVR is synthetic, limited to CLEVR-derived scenes, four discrete attributes, and single-object edits. Baseline evaluations cover only a subset of object-centric, generative, or VLM systems.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Each ingredient is an object, and each object has attributes like color, size, and shape. EditCLEVR is like a recipe that tells you how to change one ingredient's attribute without altering the others. It evaluates whether you can correctly change the color or size of an ingredient while keeping other ingredients stable.
ELI14 Explained like you're 14
Hey, imagine you're playing a game where you can change the color or size of one character without affecting others. EditCLEVR is like a tool that helps you evaluate if those changes are correct. It's like a super cool editor that lets you see the effects of your changes!
Glossary
CLEVR-style renders
A technique for generating synthetic scenes used in image rendering.
Used to generate scene pairs in EditCLEVR.
Object-centric representation
A method of representing scenes through objects and their attributes.
EditCLEVR evaluates the faithfulness of this representation.
Semantic edit
A meaningful modification of object attributes in a scene.
EditCLEVR evaluates object-centric representations under semantic edits.
Scene-Graph Intervention Accuracy (SGIA)
A metric assessing whether object-attribute scene graph changes match the intended intervention.
Used in EditCLEVR to evaluate semantic faithfulness.
Compositional OOD transfer
Evaluating a model's ability to transfer out-of-distribution compositions.
Emphasized by EditCLEVR.
Open Questions Unanswered questions from this research
- 1 How can EditCLEVR's evaluation framework be extended to natural images?
- 2 How to address the complexity of multi-object edits?
Applications
Immediate Applications
Object-centric representation evaluation
Helps researchers evaluate object-centric representations under semantic edits.
Long-term Vision
Natural image evaluation
Extend EditCLEVR to natural images to evaluate representation stability and faithfulness.
Abstract
Object-centric learning aims to represent scenes as objects whose properties can be reused in new combinations. Existing evaluations usually score segmentation, single-image factor prediction, or downstream accuracy, but these tests do not directly ask whether a per-object representation behaves correctly under a controlled semantic edit. We introduce EditCLEVR, a paired-scene intervention benchmark in which each example contains a before/after pair of CLEVR-style renders with the same object indices and scene layout, and either exactly one known attribute change on one known object or a no-edit re-render for drift measurement. The protocol includes probe-free diagnostics for representation-change localization and stability, together with probe-decoded semantic faithfulness metrics that test whether the predicted scene change matches the intended intervention across in-distribution and compositional out-of-distribution (OOD) suites, allowing code-space movement and decoded object-attribute correctness to be evaluated separately. We introduce the semantic metric Scene-Graph Intervention Accuracy (SGIA), which requires the full after-scene prediction to be correct and the only predicted before-to-after semantic change to be the intended object-factor edit. We also establish Delta-SGIA as a companion diagnostic that checks the single-site change pattern without requiring the full after-scene graph to be correct. Baseline evaluations on ground-truth-mask backbones, learned-slot models, SAM 2 + frozen-ViT models, and one mask-feature hybrid indicate that CoGenT-OOD-core degradation can persist under ground-truth instance masks, that mask source accounts for part but not all of native performance, and that locality or stability alone can overstate semantic faithfulness. Code is available at https://github.com/torux-bughunter/EditCLEVR.