Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
FWS builds FaithfulQA from six VQA benchmarks; 60K faithful SFT raises RL accuracy and stabilizes visually grounded reasoning.
Key Findings
Methodology
Faithful Warm-Start (FWS) curates data from AI2D, TextbookQA, PlotQA, ScienceQA, ChartQA, and Geo3K. Each trajectory is decomposed into Visual Grounding, Question Requirement, Intermediate Inference, and Final Prediction units. The model first receives SFT on causally screened traces, then RL with sparse answer rewards. Qwen3VL-8B-Thinking serves as the judge for purification.
Key Results
- Without RL, 20K-SFT reaches 79.00% average accuracy versus 70.91% for the base model, an 8.09-point gain. It obtains 68.94% on AI2D, 70.77% on PlotQA, and 86.56% on ScienceQA, showing that even a relatively small faithful warm start is effective.
- With RL, 20K-SFT-RL reaches 79.51%, compared with 71.07% for Base model-RL. The 60K-SFT-RL model scores 69.36% on AI2D, 75.28% on TQA, 67.97% on PlotQA, 87.90% on SciQA, and 96.60% on ChartQA.
- Reward curves show direct RL rising slowly and fluctuating around a low plateau. In contrast, 60K-SFT-RL has an initial adaptation phase, then rises sharply and converges to a higher, more stable plateau, indicating a better optimization regime.
Significance
The paper reframes a central multimodal-RL problem: success depends not only on reward design but also on policy initialization. It explains how answer-only rewards can favor language priors, dataset shortcuts, and post-hoc rationalization over image evidence. FWS gives a model an explicit route from observation to answer before RL begins. This is relevant to auditable VQA, educational systems, robotics, and safety-critical multimodal assistants that must justify decisions with observable evidence.
Technical Contribution
FWS combines structured reasoning units, unit-level causal intervention, and VLM-based judging. For a correct trace with N units, each unit is removed once and the model is asked to answer again; num_correct_to_wrong measures whether removal changes a correct prediction into an incorrect one. Traces must score 2 on visual_grounding, unit_causality, non_contradiction, and faithfulness, with causal evidence greater than zero. Unlike answer-only RL, the pipeline explicitly selects traces that are correct for grounded reasons.
Novelty
The main novelty is not a new RL objective, but a causal warm-start regime before RL. Whereas related methods often modify rewards or directly apply Visual-RFT, FWS treats the initial policy as an optimization precondition and operationalizes faithfulness through intervention-based evidence. This provides a concrete data-selection mechanism for separating grounded reasoning from fluent but redundant narration.
Limitations
- Causal necessity is approximated by re-querying a VLM after deleting text units. Results may depend on prompt format, model bias, stochasticity, and context disruption; therefore they are not equivalent to human-level causal understanding.
- Experiments center on Qwen3-VL-2B-Instruct with Qwen3VL-8B-Thinking as judge and six mostly static VQA benchmarks. Generalization to open-ended images, video, embodied interaction, and other model families remains untested.
- The paper emphasizes accuracy and reward curves but does not provide a broadly standardized numerical benchmark for visual faithfulness or a detailed accounting of the substantial intervention-time compute.
Future Work
The authors propose incorporating SFT-learned reasoning dynamics into RL reward modeling, replacing purely outcome-level supervision with process-level rewards for visual grounding, causal coherence, and logical consistency. Further work should test larger and independent models, video and interactive tasks, cheaper intervention estimators, cross-model judges, and defenses against evaluator-aware reward hacking.
AI Executive Summary
Vision-language models are moving from image description toward multi-step reasoning. Yet direct reinforcement learning often evaluates only the final answer. A model can therefore exploit language priors or dataset shortcuts, produce a fluent explanation, and still fail to use the image. This mismatch between answer-level reward and process faithfulness is a major source of instability in multimodal RL.
Peng and colleagues propose Faithful Warm-Start (FWS). From 137,568 examples across six VQA benchmarks, they generate traces organized into visual evidence, question requirements, intermediate inference, and final prediction. They retain correct traces, delete each reasoning unit in turn to test causal necessity, and use Qwen3VL-8B-Thinking to score visual grounding, unit causality, non-contradiction, and overall faithfulness. The resulting 60K FaithfulQA examples first supervise Qwen3-VL-2B-Instruct; a separate 20K pool is then used for RL.
The gains are substantial. Without RL, 20K-SFT reaches 79.00% average accuracy, versus 70.91% for the base model. After RL, 20K-SFT-RL reaches 79.51%, while direct Base model-RL reaches 71.07%. The 60K-SFT-RL model scores 96.60% on ChartQA and 87.90% on ScienceQA. Reward dynamics also improve: direct RL saturates at a low, noisy plateau, whereas faithful initialization produces a short adaptation period followed by rapid convergence to a higher, steadier plateau. The study argues that initialization is a prerequisite for reliable VLM RL, although its intervention test remains model-dependent and its evidence is limited mainly to static VQA.
Deep Analysis
Background
VLMs have advanced across VQA, chart understanding, science, and geometry, while Qwen3-VL and methods such as Visual-RFT show the promise of RL for reasoning. However, sparse rewards usually check only final correctness. A model may therefore exploit language priors or dataset bias. The unresolved issue is not merely whether the answer is right, but whether the visual evidence actually caused the answer.
Core Problem
Given an image, question, and often candidate answers, the model should derive its response from observable evidence. Direct RL optimizes the terminal reward but does not supervise intermediate grounding. Weakly grounded initialization can consequently amplify shortcuts, produce verbose post-hoc explanations, and lead to low, unstable reward plateaus. The challenge is to identify reasoning traces that are both correct and causally useful before sparse-reward optimization.
Innovation
FWS contributes three linked ideas. First, FaithfulQA represents reasoning as four functional unit types rather than undifferentiated text. Second, unit-level causal intervention deletes one unit at a time and computes num_correct_to_wrong, filtering out decorative explanations. Third, Qwen3VL-8B-Thinking applies four strict scores—visual grounding, unit causality, non-contradiction, and faithfulness—with all scores required to equal 2. Thus, the contribution is a causal data gate for RL initialization, not simply more supervised examples.
Methodology
- �� Input construction: 137,568 image-question examples come from AI2D, TextbookQA, PlotQA, ScienceQA, ChartQA, and Geo3K.
- �� Trace rollout: a VLM reasons inside <think> tags and answers inside <answer> tags; the trace is split into Visual Grounding, Question Requirement, Intermediate Inference, and Final Prediction.
- �� Correctness pre-filter: 121,866 correct trajectories are retained.
- �� Intervention: for a trace with N units, remove each unit once, retain the original image and question, and re-query the model. A positive num_correct_to_wrong indicates functional necessity.
- �� Judge filtering: Qwen3VL-8B-Thinking assigns four 0–2 scores; all must be 2 and the causal metric must exceed zero. Duplicate candidates are resolved by causal evidence and alignment.
- �� Training split: 60K examples are used for SFT, while 20K non-overlapping examples are reserved exclusively for RL; original benchmarks remain evaluation-only.
Experiments
The backbone is Qwen3-VL-2B-Instruct and the judge is Qwen3VL-8B-Thinking. Comparisons include the base model and 20K, 40K, and 60K SFT checkpoints, with and without subsequent RL. Evaluation covers AI2D, TQA, PlotQA, ScienceQA, and ChartQA. The main metrics are benchmark accuracy, average accuracy, and reward trajectories. RL reward design and training settings are shared to isolate the effect of warm-start initialization.
Results
Without RL, average accuracy is 70.91% for the base model, 79.00% for 20K-SFT, 78.57% for 40K-SFT, and 78.93% for 60K-SFT. With RL, the corresponding figures are 71.07%, 79.51%, 79.24%, and 79.42%. The 60K model reaches 96.60% on ChartQA and 87.90% on ScienceQA. Gains are already clear at 20K, while larger SFT scales approach saturation on most tasks, suggesting quality matters more than unlimited volume.
Applications
FWS is directly relevant to evidence-traceable VQA, educational diagram analysis, scientific tutoring, and geometry solving. Organizations can construct domain-specific faithful traces, perform SFT, and then apply task-level RL. Industrial inspection and robotics are promising extensions, but they require temporal evidence, sensor grounding, action feedback, and evaluation in real environments rather than static benchmark images alone.
Limitations & Outlook
The intervention pipeline requires multiple VLM calls per reasoning unit, so cost grows with trace length. Deleting text can also disrupt context rather than isolate a single cause. The judge may share biases with the trained model, and a perfect-score threshold may reduce coverage. The study uses one Qwen family and static datasets; open-ended answers, video, multilingual settings, embodied tasks, and independent faithfulness metrics remain open. Future process rewards may address these gaps.
Plain Language Accessible to non-experts
Imagine a student entering a picture-based quiz where only the final answer earns points. The student may memorize common patterns, guess from the wording, and then invent a convincing explanation afterward. That is what direct answer-reward training can encourage in an AI: fluent success without genuine attention to the picture.
FWS changes the lesson plan. The student must first point to what is visibly present, explain what the question asks, connect the facts through intermediate steps, and only then give an answer. The teacher then removes one sentence at a time and repeats the question. If removing a sentence changes a correct answer into a wrong one, that sentence was genuinely useful; if nothing changes, it may have been decoration. Another picture-capable judge checks whether the explanation matches the image, whether every step is needed, and whether the story contradicts itself.
This makes the AI learn “earn points from evidence” before it learns “earn points quickly.” With 20K such examples, average performance rose from 70.91% to 79.00%; after reinforcement learning it reached 79.51%, compared with 71.07% for direct RL. The method is promising, but repeated checking is expensive and static pictures are not the same as real-world video or robots.
ELI14 Explained like you're 14
Think of an AI playing a picture-answering game. The game only checks whether the final answer is correct, not whether the AI actually looked at the picture. So the AI might guess using familiar wording, like a gamer exploiting a shortcut. It can sound super confident and still be wrong for the right reasons—or right for the wrong reasons!
The researchers give it six kinds of challenges: diagrams, charts, science questions, geometry, and textbook problems. The AI must explain four things: what it sees, what the question wants, how the clues connect, and the final answer. Then the researchers delete one explanation step and ask again. If the answer breaks, that step mattered. If nothing changes, it was probably just fancy text.
A stronger AI acts like a referee. It checks whether the explanation really matches the image, whether each step is necessary, and whether the steps disagree. Only the best examples teach the model before it starts reinforcement learning. With 20K examples, its average score jumped from 70.91% to 79.00%. After RL, the warm-start version scored 79.51%, while direct RL reached only 71.07%.
Pretty impressive, right? But there is a catch: checking every sentence takes lots of computer work, and the tests mostly use still images. A robot watching a moving street or making a real decision is much harder. The next challenge is making AI prove it is actually looking, not just writing like it is!
Glossary
Faithful Warm-Start
A supervised initialization stage that teaches a model visually grounded reasoning before reinforcement learning. Technically, it constrains the starting policy toward traces with verified causal dependence on image evidence.
The paper’s central training strategy.
FaithfulQA
A curated dataset of high-confidence multimodal reasoning traces derived from six VQA benchmarks. Each trace contains observations, question requirements, intermediate inference, and a final prediction.
Used for SFT, not final benchmark evaluation.
Unit-level causal intervention
An evaluation procedure that removes one reasoning unit and measures the change in the model’s answer. It approximates whether that unit is functionally necessary for the decision.
Used to compute num_correct_to_wrong.
Visual grounding
The degree to which a response relies on observable image content such as objects, text, positions, or relations. It contrasts with answering from language priors alone.
A reasoning-unit category and judge dimension.
Sparse answer-level reward
A reward supplied mainly according to final-answer correctness, with little or no direct feedback on intermediate reasoning. Such rewards can permit shortcuts and post-hoc rationalization.
The failure mode motivating FWS.
num_correct_to_wrong
The number of cases in which deleting a reasoning unit changes a correct prediction into an incorrect one. A value above zero is treated as evidence of causal usefulness.
A FaithfulQA selection criterion.
Open Questions Unanswered questions from this research
- 1 How can visual causal faithfulness be evaluated without repeatedly re-running a VLM for every reasoning unit? Cheaper intervention estimators, external evidence labels, or verifiable programs are needed.
- 2 Does FWS transfer to video, open-ended answers, multilingual tasks, and embodied decisions? The six mostly static benchmarks cannot establish this.
- 3 Can independent judges avoid inheriting the target model’s biases? Robust, calibrated process evaluation is still needed to resist evaluator-aware reward hacking.
Applications
Immediate Applications
Educational diagram and science tutoring
Education platforms can generate grounded traces for AI2D-, ScienceQA-, and ChartQA-like tasks, then apply SFT followed by answer-level RL. Students and teachers receive both an answer and image-linked evidence, improving auditability and reducing hallucinated explanations.
Industrial visual inspection
Inspection systems can organize defect location, appearance, and domain rules into four-stage traces, then use intervention filtering to remove irrelevant descriptions. Domain image standards and human audits remain prerequisites; expected benefits include clearer alarms and easier fault tracing.
Long-term Vision
Trustworthy multimodal agents
An extended FWS could train robots to cite visual and sensor evidence before acting in video environments. Major obstacles include real-time intervention cost, causal reasoning over continuous states, reliable process rewards, and validation under distribution shift.
Abstract
Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs). However, directly applying RL to rollout multimodal reasoning can lead to instability, due to the exploitation of language priors, the neglect of visual evidence, and the generation of reasoning traces that are fluent yet not visually grounded. The question arises: Can initially steer the policy toward visually faithful reasoning regime before applying reinforcement learning? To this end, we propose a Faithful Warm-Start (FWS) strategy that first curates samples with explicit vision-language causal relationships from six general VQA benchmarks to construct the FaithfulQA dataset, where each of the image-question pairs gains a certain degree of visual observations, question requirements, commonsense knowledge, domain knowledge, and the final answer. Subsequently, a VLM-based judge is employed to further purify the dataset, ensuring strong causal consistency and visual faithfulness. This warm-start stage equips the model with the capability to understand causally grounded vision-language patterns before subsequent RL optimization under sparse answer-level rewards. Experimental results show that such faithful supervision improves answer accuracy, stabilizes RL training, and reduces visually unsupported reasoning.