EditCLEVR benchmark tests if AI vision models truly understand object edits
New metric reveals object-centric models fail at controlled scene edits despite high accuracy...
A new paper on arXiv introduces EditCLEVR, a benchmark designed to test whether object-centric representation models truly understand compositional edits. Unlike prior evaluations that only measure segmentation or single-image accuracy, EditCLEVR presents paired before/after scenes from CLEVR-style renders where exactly one known attribute of one object changes (or no edit for drift detection). The benchmark includes probe-free diagnostics for representation-change localization and stability, plus probe-decoded semantic faithfulness metrics.
The authors propose SGIA (Scene-Graph Intervention Accuracy), which requires the full predicted after-scene graph to be correct and the only semantic change to match the intended edit. A companion metric, Delta-SGIA, relaxes the full-graph requirement to check single-site change patterns. Baseline evaluations on ground-truth-mask backbones, learned-slot models, SAM 2 + frozen-ViT models, and a mask-feature hybrid reveal critical findings: CoGenT-OOD-core degradation persists even with perfect instance masks, mask source accounts for only part of native performance, and metrics based solely on locality or stability overstate semantic faithfulness. Code is available.
- EditCLEVR uses paired before/after CLEVR renders with exactly one attribute change per object to test compositional faithfulness.
- New metric SGIA requires the full predicted scene graph to be correct and the only change to match the intended edit; Delta-SGIA offers a relaxed alternative.
- Even ground-truth masks don't prevent CoGenT-OOD-core degradation, and localization-alone metrics overstate model faithfulness.
Why It Matters
Exposes hidden failures in object-centric AI models, guiding more robust vision systems for real-world scene understanding.