SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification
SEER exposes query-specific evidence for frozen VLMs, improving frozen GQA-Train900 accuracy by 3.94 points over Full.
Key Findings
Methodology
SEER is a training-free inference interface for frozen vision-language models. A grounder first localizes the queried subject and object while candidate relations are hidden. The system then constructs Full-image, role-explicit S/O-Marker, and selectively eligible Geometry evidence. For each visual state, the VLM computes length-normalized option log-probabilities and margins. Equation (2) chooses Geometry when its score exceeds τ=0.5, otherwise the higher-margin visual state. Equation (3) optionally applies exact inverse consistency after swapping subject and object.
Key Results
- On the image-disjoint, pre-frozen GQA-Train900 test, Qwen3-Instruct and Qwen3-Thinking gain 3.11 and 4.78 points over Full, respectively. The pooled gain is 3.94 points with a 95% interval of [2.17, 5.72].
- Across all 2,434 filtered EmbSpatial pair-relation questions, Qwen3-Instruct, Qwen3-Thinking, and InternVL3.5 improve over Full by 4.35, 5.09, and 11.79 points, respectively, indicating transfer across model families.
- Controls identify Marker as the dominant intervention: it supplies about a 2.18-point initial gain in development pools. Reciprocal refinement adds about 1.36 points; on GQA-Train900 it triggers on 10.61% of model-example pairs, with 40 fixes versus 18 breaks.
Significance
The paper reframes spatial VLM errors as failures of evidence selection and entity binding, not merely missing spatial vocabulary. A model may recognize both objects yet choose the wrong instance, reverse the roles, or rely on an ambiguous global image. SEER improves this interface without training, weight updates, or an external detector, offering a low-cost intervention for VQA, embodied agents, diagram interpretation, and safety-oriented visual assistance.
Technical Contribution
SEER turns a queried pair into explicit, complementary evidence states while preventing relation leakage during localization. Invalid boxes fall back to Full, preserving a common denominator; Geometry is restricted to relations supported by 2D layout, such as left/right and inside/contains. The fixed threshold τ=0.5 and deterministic selector avoid learned calibration. Subject-object exchange supplies a protocol-specific equivariance check rather than a generic confidence estimator.
Novelty
Unlike Set-of-Mark, Graph-of-Mark, or external detectors, SEER uses the frozen VLM itself to localize only the queried entities and assigns explicit S/O roles. Unlike spatially trained models, it introduces no parameters or training data. Its fundamental novelty is the leakage-controlled composition of hidden-relation grounding, role-explicit local evidence, retained global context, selective geometry, and deterministic conflict resolution.
Limitations
- A semantically wrong but formally valid box can mislead Marker and Geometry; the Full fallback detects invalid coordinates, not all grounding errors.
- Two-dimensional boxes cannot reliably encode front/behind and cannot distinguish above/below/on from vertical displacement alone, so Geometry abstains on these families.
- Reciprocal refinement requires exact inverse mappings and suitable answer protocols; it is not directly applicable to open-ended generation or relations such as near/far.
Future Work
Future work should combine self-grounding with segmentation, depth, temporal tracking, and calibrated uncertainty. Broader evaluation is needed on VSR, SpatialSense, multilingual queries, open-ended generation, and real robotic scenes. A lightweight learned evidence selector could improve reliability, but it should preserve relation-hidden localization, deterministic auditing, and the safe Full fallback.
AI Executive Summary
Vision-language models can recognize a red mug and a blue book yet still answer their spatial relation incorrectly. The failure often occurs before reasoning: the model selects the wrong instance, reverses subject and object roles, or relies on an ambiguous global view. SEER, or Self-grounded Evidence for Entity-Relation Reasoning, treats the problem as evidence construction. During localization, it hides candidate relations so the grounder cannot choose boxes by reading the answer options.
SEER constructs three complementary evidence states. Full preserves the original scene; a padded S/O-Marker crop highlights the queried pair and explicitly labels subject and object; Geometry supplies a thresholded box-based cue when two-dimensional layout is sufficient. The same frozen VLM scores Full and Marker using option log-probability margins. When exact inverse relations are guaranteed, SEER swaps the roles and changes the forward decision only if exactly one state satisfies reciprocal consistency. Invalid grounding returns Full.
On the image-disjoint GQA-Train900 test, SEER improves over Full by 3.94 points, with a 95% interval of [2.17, 5.72]. On 2,434 EmbSpatial questions, gains range from 4.35 to 11.79 points across three models. Matched controls show that local refocus and explicit roles drive most of the improvement, while reciprocal consistency is a smaller protocol-dependent refinement. The broader lesson is that better evidence interfaces can outperform simply asking a model to reason longer.
Deep Analysis
Background
VSR, What’sUp, GQA, ARO, and SpatialSense show that VLMs often recognize entities but fail to bind spatial relations. Set-of-Mark and Graph-of-Mark expose regions, while SpatialVLM and Spatial-RGPT use stronger training or spatial representations. SEER instead asks whether a frozen model can improve through a better inference-time evidence interface.
Core Problem
For x=(I,s,o,A), the goal is to select the relation r(s,o) from answer set A. Multiple instances, ordered subject-object semantics, and global-image ambiguity create errors. If grounding sees candidate relations, it risks answer leakage; if the model sees only a crop, useful context may disappear.
Innovation
SEER contributes four linked ideas: relation-hidden self-grounding; a query-specific crop with explicit S/O roles; complementary Full, Marker, and selectively admitted Geometry states; and exact reciprocal checking through subject-object exchange. The contribution is their controlled composition, not a new crop or glyph in isolation.
Methodology
- �� Grounding: frozen G receives image and entity names, excluding answer options and relation labels, and outputs normalized boxes bs and bo.
- �� Validation: boxes must have positive area and remain inside the clipped image; otherwise the method falls back to Full.
- �� Marker: a padded union crop draws a red S box around the subject and a blue O box around the object.
- �� Geometry: box centers, extents, or coverage yield a candidate only for supported relation families and score c≥τ, with τ=0.5.
- �� Scoring: for each visual state t, the frozen M computes option log-probabilities ℓi(t)(a) and margin m=best score−second-best score.
- �� Forward decision: Geometry has priority when eligible; otherwise Marker wins only with a larger margin, and ties favor Full.
- �� Refinement: when Geometry abstains and Full/Marker disagree, swap S/O and retain a changed state only if exactly one satisfies the inverse map ι(r).
Experiments
The study uses 11,070 GQA and 9,702 Visual Genome development examples, a frozen GQA-Train900 test with 900 unique images, and 2,434 filtered EmbSpatial questions from 276 scenes. Models include Qwen3-VL-8B-Instruct/Thinking, Qwen2.5-VL-7B-Instruct, R1-Onevision-7B, and InternVL3.5-8B-HF. Full, Crop, Marker, Forward, and SEER are compared using accuracy, paired error counts, McNemar tests, and 20,000 clustered bootstrap repetitions.
Results
On GQA-Train900, Qwen3-I rises from 75.33 to 78.44 and Qwen3-T from 70.56 to 75.33; the pooled gain is 3.94 points. On complete development pools, gains are 3.03 and 4.95 points for Qwen3-I and Qwen3-T. Balanced cross-model development gains 4.26 points overall. EmbSpatial gains are 4.35, 5.09, and 11.79 points. Marker contributes most; reciprocal refinement adds a smaller positive increment.
Applications
SEER can augment frozen VQA, visual search, assistive description, and document-understanding systems without retraining. It is especially useful when questions refer to one of several similar objects. Robotics and inspection systems can combine Full context with Marker roles and coarse Geometry, provided the model exposes reliable boxes and option scores.
Limitations & Outlook
The method depends on self-grounding, and valid-looking wrong boxes remain dangerous. Two-dimensional geometry cannot represent depth, occlusion, or many open-world relations. The evidence is strongest in controlled relation classification, not necessarily in free-form generation. Future systems should add segmentation, depth, uncertainty estimation, temporal evidence, and evaluations in dynamic multilingual environments.
Plain Language Accessible to non-experts
Imagine a warehouse clerk asked, “Is the red mug left or right of the blue book?” The warehouse contains several mugs and books. A clerk who simply looks at the whole room may pick the wrong pair, even while recognizing every object. SEER changes the paperwork before asking for the answer. It hides the possible answers and first tells the clerk exactly which mug and which book to find, preventing the clerk from choosing objects that conveniently support a guessed relation.
The clerk receives three sheets. One shows the entire warehouse, preserving useful surroundings. Another is a zoomed view containing the two selected objects, with S and O labels that say who is the subject and who is the reference object. The third is a simple map showing their positions. The clerk uses the map only when it is clear enough; otherwise, the visual sheets decide.
If the two visual sheets disagree, the clerk asks the reversed question. If “mug left of book” becomes “book right of mug” in exactly one sheet, that sheet earns the decision. Otherwise, SEER does not force a correction. The experiments show that preparing the right evidence is often more useful than asking the clerk to think harder from the same confusing view.
ELI14 Explained like you're 14
Think of a video-game map with several red characters and blue treasure chests. The question is: “Is the red character left or right of the blue chest?” A normal AI may know what red, character, blue, and chest mean, but still look at the wrong pair. Oops! SEER acts like a helpful teammate who first hides the answer choices and circles only the two objects the question names.
Then it gives the AI three views: the whole map, a zoomed picture with a red S label for the subject and a blue O label for the object, and a tiny position map. S and O matter because “A is left of B” is not the same as “B is left of A.” The AI now gets both a close-up and a clear reminder of who is who.
If two views disagree, SEER swaps the roles and checks whether the answer should flip. Left should become right, and inside should become contains. It changes the answer only when one view follows this rule and the other does not. No new training is needed—just better instructions and evidence.
On GQA-Train900, SEER scored 3.94 points higher than using the full image alone. On EmbSpatial, three models improved by 4.35 to 11.79 points. Still, it can circle the wrong object, and a flat map cannot truly show what is in front. So it is a smart teammate, not magic!
Glossary
Self-grounding
The model locates image regions corresponding to named entities. In SEER, this happens without seeing candidate relations.
Produces subject and object boxes bs and bo.
Full Image Evidence
The unchanged image and original question, preserving global context. It is always available and serves as the fallback.
Primary baseline and safety state.
S/O-Marker
A local view that marks the subject with S and the object with O. It combines local refocus with explicit role assignment.
Main visual intervention in SEER.
Geometry Evidence
A deterministic relation candidate derived from box centers, extents, or coverage. It is thresholded and abstains when 2D layout is insufficient.
Enabled with fixed τ=0.5.
Reciprocal consistency
A relation should transform to its exact inverse when subject and object are exchanged, such as left-of/right-of. SEER uses this only for eligible conflicts.
Optional refinement in Equation (3).
Option-score margin
The difference between the highest and second-highest option log-probabilities. It is an ordering cue, not a calibrated probability of correctness.
Selects between Full and Marker in Equation (2).
Open Questions Unanswered questions from this research
- 1 How can systems detect semantically wrong but geometrically valid boxes? The current fallback catches malformed coordinates, not every identity or referent error.
- 2 How should depth, occlusion, and temporal structure be represented? Two-dimensional boxes cannot reliably solve front/behind or distinguish above, below, and on.
- 3 Will SEER remain robust in open-ended generation, multilingual reference, and real robots? Larger external evaluations and uncertainty-aware evidence are needed.
Applications
Immediate Applications
Frozen-VLM visual question answering
A VQA service can run relation-hidden grounding, generate an S/O-Marker view, and score it alongside the original image. It needs box output and option likelihoods, but no parameter updates; the expected benefit is fewer multi-instance and role-binding errors.
Assistive image and document understanding
Accessibility tools can highlight the two objects named in a user’s question, such as chart elements, people, or furniture. Full preserves context while Marker clarifies exactly which entities should be compared.
Long-term Vision
Evidence interfaces for embodied agents
A robot could convert instructions such as “take the cup left of the box” into explicit subject-object evidence before acting. The vision stack would need depth, dynamic tracking, calibrated uncertainty, and real-time grounding to make this safe and reliable.
Abstract
Spatial relation questions require a model to identify the queried subject and object before comparing their layout. Yet a VLM can recognize both entities and still answer from the wrong instance or an ambiguous global view. We ask whether making query-specific evidence explicit can mitigate this failure and propose SEER (Self-grounded Evidence for Entity-Relation Reasoning), a training-free inference-time evidence interface for frozen VLMs. SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence. For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation. On an image-disjoint GQA-Train900 test frozen before model scoring, SEER pools to +3.94 [2.17,5.72] over Full; the gain remains positive under label-independent grounding-order counterbalancing and on the 535 rows whose entity names are unique. The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models. Matched controls separate local refocus from role-explicit conditioning. These results establish query-specific evidence construction as the principal intervention, with reciprocal consistency as a smaller protocol-specific refinement.