Negation-Aware Test-Time Adaptation for Vision-Language Models
NEAT adapts only normalization layers, resolving VLM negation shifts with under 0.01% trainable parameters.
Key Findings
Methodology
NEAT updates only normalization layers in the text encoder during inference. It combines entropy refinement with negation-separated candidate selection, semantics reversion learning using reversed captions as hardest negatives, and textual dual-concept debiasing. Llama-3-8B decomposes each negated caption into positive and excluded components, while a triplet objective brings the original negation closer to its affirmative form and pushes it away from its semantic reversal.
Key Results
- On CIFAR10, the MMD between normal and negation-conditioned text is 1.57, despite semantic consistency; the MMD between negation-conditioned and reversed-negation text is only 0.08, despite opposite meaning.
- Across COCO, VOC2007, MSR-VTT, and CheXpert, spanning image, video, and medical tasks, NEAT matches or surpasses expensive post-training baselines while using below 0.36‰ of the unlabeled data and 0.014‰ of trainable parameters.
- The adapted layers generalize directly to unseen datasets and tasks. The paper reports comparable or superior performance to state-of-the-art post-training approaches, although the supplied text does not provide complete task-level scores.
Significance
The paper reframes VLM negation failure from a simple shortage of negative examples into a distribution-alignment problem. This offers a lower-carbon alternative to retraining on NegFull’s more than 70 million synthetic pairs or NegationCLIP’s 229k augmented pairs. The approach is relevant to medical search, video retrieval, robotics, and safety-critical queries where excluding a concept is as important as detecting one.
Technical Contribution
NEAT treats normalization parameters as distribution-related variables suitable for test-time adaptation. Its candidate score S=(tP·v)(1-α[tN·v]+) rewards positive matching while penalizing excluded concepts. Its semantics-reversion loss compares the reversed caption only with the target and hardest visual negative, avoiding the collapse induced by broad negative InfoNCE. The triplet textual loss directly regularizes the geometry of affirmative, negated, and reversed texts.
Novelty
According to the authors, NEAT is the first test-time adaptation framework specifically designed for VLM negation understanding. Unlike NegCLIP, CoNCLIP, NegBench, and NegationCLIP, which rely primarily on targeted negatives or large-scale post-training, it separately addresses consistent-semantic shift and false consistency between semantically opposite captions through lightweight online objectives.
Limitations
- The framework depends on LLM-generated decomposition and reversal. Nested negation, anaphora, ellipsis, domain terminology, or ambiguous scope can produce incorrect supervision and destabilize online adaptation.
- The reported evaluation centers on CLIP-style models and four datasets. The supplied paper text lacks complete per-task scores and does not fully establish robustness under tiny batches, long streaming sequences, severe domain shift, or changing negation syntax.
Future Work
Future research should develop differentiable or self-contained negation parsing, multilingual and nested-negation benchmarks, continual adaptation with catastrophic-forgetting control, and extensions to MLLMs such as GPT-4o. Complete task-level scores, latency, energy, calibration, and safety evaluations would clarify deployment readiness.
AI Executive Summary
Vision-language models can align images with words, yet often fail when a query says “without,” “not,” or “excluding.” Han and colleagues show that the problem is not merely missing vocabulary. On CIFAR10, normal and negation-conditioned captions have an MMD of 1.57 although they are semantically compatible; negation-conditioned and reversed captions have an MMD of only 0.08 although they mean opposite things. The model therefore behaves like a bag-of-words matcher.
The authors propose Negation-Aware Test-Time Adaptation, or NEAT. Llama-3-8B separates each query into a positive component and an excluded component, then produces a reversed caption. NEAT selects visual candidates that match the positive content but avoid the negative content, treats the reversed caption as a hardest negative, and applies a triplet textual objective. Only normalization layers are updated, rather than the full VLM.
Experiments cover COCO, VOC2007, MSR-VTT, and CheXpert, including image retrieval, multiple-choice reasoning, video retrieval, and medical classification. NEAT matches or exceeds costly post-training baselines while using below 0.36‰ of unlabeled data and 0.014‰ of trainable parameters; the paper also summarizes the method as using under 0.01% trainable parameters. The approach is promising for exclusion-based search, but its dependence on LLM parsing, limited published score detail, and uncertain behavior under complex or continual negation remain important open issues.
Deep Analysis
Background
CLIP and ALIGN established strong transfer from large-scale image-text pretraining, but web captions are predominantly affirmative. CREPE, CC-Neg, and NegBench exposed compositional failures involving negation. Data-centric responses include NegBench’s over 70 million synthesized pairs and NegationCLIP’s 229k augmented pairs. These methods help, but their data and compute costs motivate a lightweight alternative.
Core Problem
A VLM must identify present concepts while excluding absent or prohibited ones. The paper defines two coupled failures: affirmative and correctly negated captions drift apart, while a negated caption and its semantic reversal remain too close. This makes retrieval, multiple-choice selection, and classification unreliable even when the underlying visual concept is familiar.
Innovation
NEAT contributes three linked ideas. First, negation-separated candidate selection makes entropy minimization more reliable than raw nearest-neighbor matching. Second, a reversed negation caption becomes the hardest negative, eliminating false distributional consistency without pulling every unrelated image together. Third, textual triplet debiasing directly shapes the three-way geometry. This differs from NegCLIP, CoNCLIP, NegBench, and NegationCLIP, which mainly add negative data or perform post-training.
Methodology
- �� Input: an unlabeled online batch of images and negated captions.
- �� Parsing: Llama-3-8B outputs positive tP, excluded tN, and reversed caption t̄.
- �� Selection: choose v using S=(tP·v)(1-α[tN·v]+).
- �� Entropy refinement: minimize candidate-matching entropy with temperature τ1.
- �� Reversion learning: compare each target with its reversed-caption hardest visual negative using τ2.
- �� Text debiasing: minimize λ||t̂-tP||²+(2-||t̂-t̄||²).
- �� Update: optimize only normalization parameters in the text encoder.
Experiments
Datasets are COCO (Retrieval-Neg 5,000; MCQ-Neg 5,914), VOC2007 (5,032), MSR-VTT (1,000 retrieval and 1,000 MCQ), and CheXpert (2,352 affirmative and 616 negative binary examples). Comparisons include post-training and TTA baselines across image, video, and medical domains. The study also evaluates component effects and transfer to unseen datasets and tasks.
Results
The CIFAR10 MMD values, 1.57 and 0.08, support the proposed dual-shift diagnosis. On the four benchmark families, NEAT reaches or exceeds post-training baselines with below 0.36‰ of unlabeled data and 0.014‰ of trainable parameters; the abstract rounds this to under 0.01%. Adapted layers transfer to unseen datasets and tasks, suggesting that distribution correction rather than full semantic retraining is effective.
Applications
Radiology systems could search for nodules without malignant features; drones could retrieve roads without ice; and video or product-search systems could find content containing a target while excluding distractors. Deployment requires online unlabeled batches and reliable decomposition of the query, potentially through a lightweight external LLM.
Limitations & Outlook
External LLM parsing adds latency, cost, and error propagation. The method assumes that positive and excluded concepts can be separated cleanly, which is difficult for nested scope, open-world unknowns, and implicit negation. Normalization-only adaptation may be unstable with tiny batches, severe shifts, or long streams. The supplied paper text does not provide full per-dataset scores, energy measurements, or extensive calibration and safety analysis.
Plain Language Accessible to non-experts
Imagine a warehouse clerk receiving the request: “Find a bus, but no pedestrian nearby.” A keyword-driven clerk may notice both “bus” and “pedestrian” and choose any photo containing them. The clerk has recognized the words but ignored the instruction to exclude something.
NEAT gives the clerk three simple routines. First, an assistant rewrites the request into “look for this” and “avoid that.” The clerk then selects photos that resemble the desired object while not resembling the forbidden one. Second, the assistant creates the most confusing opposite request—swap the desired and forbidden objects. The clerk must reject that near-looking mistake. Third, the clerk adjusts only a few sorting rules instead of retraining the whole warehouse staff.
This saves examples, time, and electricity. NEAT was tested on COCO, VOC2007, MSR-VTT, and CheXpert, covering images, videos, and medical scans. It uses under 0.01% trainable parameters while matching or outperforming costly retraining approaches. The remaining weakness is that a language assistant can misunderstand complicated wording.
ELI14 Explained like you're 14
Picture a game where you ask a bot to find “a treasure chest in a room with no enemies.” A keyword bot may see both “treasure” and “enemy” and think the room is a great match. It knows the words, but not the tiny word “no.” That is the surprising weakness many vision-language models have.
NEAT works like three upgrades. The first splits the mission into “find treasure” and “avoid enemies.” The second shows the bot a tricky opposite mission—“find enemies, but no treasure”—so it learns that the two missions are not the same. The third moves the correct descriptions closer together and pushes opposite descriptions apart. Nice and simple, right?
The cool part is that NEAT does not rebuild the whole robot. It changes only a very small set of settings while the robot is taking its test. On COCO, VOC2007, MSR-VTT, and CheXpert, it competes with methods trained on huge extra datasets, using less than 0.01% trainable parameters.
But it is not magic. A language model helps split the sentence, so confusing wording can fool it. What about double negatives, slang, or a never-seen object? Those are exactly the next levels researchers need to beat!
Glossary
Vision-Language Model
A model that maps images and text into comparable representations. Technically, it uses visual and textual encoders for cross-modal similarity.
CLIP is the main representative model analyzed and adapted.
Test-Time Adaptation
Updating a small part of a pretrained model using unlabeled test inputs at deployment time. It avoids full retraining on the original training set.
NEAT adapts text-encoder normalization layers online.
Dual-Concept Shift
A paired phenomenon in which semantically compatible affirmative and negated texts separate, while semantically opposite negated and reversed texts overlap.
CIFAR10 MMD values of 1.57 and 0.08 quantify it.
Negation Separation
Decomposing a negated sentence into content to preserve and content to exclude. It converts difficult negative semantics into simpler affirmative matching problems.
Llama-3-8B produces tP and tN for candidate selection.
Reversed Contrastive Learning
Using a semantic reversal as a hard negative so the target image is separated from an opposite caption. It discourages accidental lexical similarity.
NEAT implements this with the semantics-reversion loss Lsr.
Maximum Mean Discrepancy
A statistical distance between two representation distributions. Larger values indicate stronger distributional mismatch.
The paper uses MMD to diagnose text-embedding shifts.
Open Questions Unanswered questions from this research
- 1 Can nested, implicit, cross-sentence, and multilingual negation be parsed reliably without an external LLM? Dedicated linguistic benchmarks and self-contained parsers are needed.
- 2 Will continual online adaptation accumulate errors or forget prior capabilities? Long streaming evaluations, tiny-batch tests, and rollback mechanisms remain necessary.
Applications
Immediate Applications
Exclusion-aware medical retrieval
Radiologists could query for findings such as nodules without malignant features. Deployment requires medically reliable parsing, unlabeled CheXpert-like batches, and human verification; the expected benefit is fewer false candidates caused by ignored negation.
Disaster and video search
Drones or media platforms could retrieve icy roads, empty streets, or other target-plus-exclusion queries. The system needs online adaptation and dependable positive/negative decomposition, making it attractive when full CLIP retraining is impractical.
Long-term Vision
Controllable multimodal agents
Robots, autonomous vehicles, and safety systems could jointly detect goals and exclude hazards. Real transformation requires robust nested-negation reasoning, open-world uncertainty handling, multilingual support, and continual-learning safeguards.
Abstract
In this paper, we study a practical but less-touched problem in Vision-Language Models (VLMs), \ie, negation understanding. Specifically, many real-world applications require models to explicitly identify what is false or non-existent, \eg, radiologists may search for images that exclude specific conditions. Despite the impressive transferability of VLMs through large-scale training, they suffer from a critical limitation that fails to handle negation. To address this challenge, existing methods attribute its root cause to the scarcity of negation training data and propose to fine-tune VLMs on massive data containing explicit negation. Undoubtedly, such data-centric solutions demand substantial data and computational resources, limiting their sustainable widespread adoption. To tackle negation in a low-carbon manner, we empirically observe that the key obstacle lies in the dual-concept shifts between the affirmation and negation distributions. Therefore, we propose a Negation-Aware Test-Time Adaptation (NEAT) method to efficiently adjust distribution-related parameters during inference. In brief, NEAT can reduce distribution shift in consistent semantics while eliminating false distributional consistency in unrelated semantics. Extensive experiments on the various negation understanding tasks verify the effectiveness of the proposed method. Remarkably, with less than 0.01\% of trainable parameters, NEAT achieves comparable or superior performance to state-of-the-art post-training approaches. Our code is available at https://github.com/hhc1997/NEAT.