CA-OPD: Confidence-Aware On-Policy Distillation for Structured Visual Prediction
CA-OPD uses teacher confidence to repair student rollouts, raising ScreenSpot-Pro by 9.50 points and OCRBench-v2 English by 6.72 points.
Key Findings
Methodology
CA-OPD trains on student-generated rollouts while using teacher confidence to gate each proposed token. It computes NLL=-logπT(ŷt|st); unsupported proposals are replaced by the teacher argmax token and written into the prefix. Replaced positions receive cross-entropy, while retained positions receive truncated top-k forward KL. A cosine schedule progressively relaxes the NLL threshold, transferring rollout control back to the student.
Key Results
- With Qwen3.5-0.8B as student and two domain-specific Qwen3.5-9B teachers, annealed CA-OPD beats OPD on all six target benchmarks. ScreenSpot-Pro reaches 45.92 versus 40.82 for OPD, while OCRBench-v2 English reaches 52.47 versus 50.79.
- Relative to the base Qwen3.5-0.8B, the abstract reports gains of 9.50 points on ScreenSpot-Pro and 6.72 points on OCRBench-v2 English. Table 1 also shows retained performance on RefCOCO at 81.90 and MMBench at 80.16.
- Ablations show that frequency alone is insufficient: random writeback obtains 40.20 on ScreenSpot-Pro, confidence gating without prefix writeback obtains 40.92, and the full method obtains 45.92. Fixed and reversed schedules reach only 44.58 and 44.36.
Significance
The work addresses a central failure mode in structured visual prediction: OCR symbols, coordinates, and GUI actions are generated sequentially, so one early mistake corrupts later contexts. CA-OPD jointly improves state visitation and token supervision. For compact VLMs, this offers a practical route to inherit grounding and recognition capabilities from larger teachers without requiring teacher inference at deployment.
Technical Contribution
The key contribution is coupling rollout behavior and supervision through one replacement indicator rt. When rt=1, the model writes back a deterministic teacher correction and applies CE; when rt=0, it retains the student token and applies teacher-distribution KL. This is combined with absolute-NLL gating and a strict-to-relaxed cosine threshold schedule, unifying prefix repair, state visitation, and knowledge transfer in one discrete autoregressive process.
Novelty
Unlike Offline KD, pure OPD, and SKD’s relative-rank acceptance rule, CA-OPD measures the teacher’s absolute support for the exact student proposal and uses the intervention outcome to select the learning objective. Its fundamental novelty is therefore not merely selective replacement, but the joint design of where to intervene, how to supervise the position, and how to return control to the student over training.
Limitations
- The evaluation focuses on GUI grounding and OCR with two domain-specialized Qwen3.5-9B teachers. It remains unclear whether absolute NLL gating is reliable for open-ended generation, long-horizon reasoning, or domains where the teacher is poorly calibrated.
- Top-k KL reduces vocabulary-transfer cost but discards tail probabilities and requires nonnegative clipping. Per-token teacher evaluation and discrete writeback also increase training memory, latency, and engineering complexity.
- The paper provides limited sensitivity analysis for threshold values, top-k size, teacher calibration, and teacher scale, although these factors may strongly affect intervention behavior.
Future Work
Future work should investigate calibrated and task-adaptive thresholds, token-type-aware intervention, teacher caching, and batched verification. Important extensions include video, tool use, long-horizon GUI action, multilingual OCR, alternative teacher architectures, and methods that make state repair cheaper or partially differentiable.
AI Executive Summary
Autoregressive vision-language models can express OCR, visual grounding, and GUI actions through one token interface, but this convenience creates a severe propagation problem: an incorrect character or coordinate becomes part of every later input. Offline KD follows stable teacher trajectories that do not match inference, while standard OPD follows student trajectories that may collapse early in training. SKD improves safety through teacher ranking, but ranking does not reveal how much probability the teacher assigns to a proposal.
CA-OPD evaluates every student proposal with teacher NLL, NLL=-logπT(ŷt|st). Low-support proposals are replaced by the teacher argmax token and written into the future prefix; supported proposals remain student-generated. Replaced positions receive direct cross-entropy, whereas retained positions receive top-k forward KL from the teacher. A cosine threshold schedule starts strict and gradually relaxes, returning control to the student as competence improves.
Using a Qwen3.5-0.8B student and two Qwen3.5-9B domain teachers, annealed CA-OPD achieves 45.92 on ScreenSpot-Pro versus 40.82 for OPD, and 52.47 versus 50.79 on OCRBench-v2 English. Random interventions, no-writeback gating, fixed schedules, and reversed schedules are all weaker. The evidence therefore attributes the gains to strategic intervention placement, actual prefix repair, progressive control transfer, and intervention-aligned supervision—not simply to more teacher corrections. Broader validation and lower-cost training remain open challenges.
Deep Analysis
Background
Pix2Seq, Donut, and Unified-IO established serialization of boxes, documents, and heterogeneous visual outputs into autoregressive sequences. Modern VLMs extend this interface to coordinate tokens and GUI actions. Classical KD transfers teacher distributions; GKD queries teachers on student rollouts; SKD interleaves teacher corrections using relative ranking. Yet structured outputs are unusually prefix-sensitive: static teacher trajectories miss inference states, while unrestricted student rollouts amplify early errors.
Core Problem
At state st=(x,y<t), the student samples ŷt from πS and uses it to construct the next state. A wrong token therefore damages both the current prediction and all subsequent contexts. Ranking only measures relative ordering and can accept a high-ranked token with weak absolute support. Moreover, rejecting a proposal identifies a concrete local error that should affect supervision. The challenge is to decide both what enters the prefix and how that position should be trained.
Innovation
First, CA-OPD replaces SKD-style ranking with absolute teacher NLL. Second, rejected proposals are deterministically repaired with teacher argmax and written back, improving future states. Third, the gate selects the objective: CE for repaired positions and top-k forward KL for retained ones. Fourth, a cosine strict-to-relaxed schedule protects early trajectories while progressively restoring student control, rather than imposing a fixed replacement rate.
Methodology
- �� The student samples ŷt∼πS(·|st;θ) from the current mixed prefix.
- �� The teacher computes NLLT,t=-logπT(ŷt|st); rt=1 when NLL exceeds τnll,s.
- �� If rt=1, the committed token is ỹt=argmaxvπT(v|st), written into the prefix; otherwise ŷt is retained.
- �� Replaced positions use LRt=-logπS(yt|st;θ); retained positions use top-k truncated KL with a zero lower bound.
- �� The sequence loss is Σmt[λCErtLRt+λKL(1-rt)LK-topkt]/Σmt.
- �� τnll follows τstart+(τend-τstart)(1-cos(πps))/2. Discrete rollout operations receive no gradient.
Experiments
The student is an SFT-initialized Qwen3.5-0.8B. Two separately fine-tuned Qwen3.5-9B models serve as grounding and OCR teachers, with examples routed by domain. Target benchmarks are ScreenSpot-v2, ScreenSpot-Pro, OCRBench-v2 English/Chinese, CC-OCR, and OmniDocBench; RefCOCO/+/g and MMBench test retention. Baselines are Offline KD, OPD, SKD, fixed-threshold CA-OPD, and annealed CA-OPD. Controlled methods share initialization, data, and optimization budget, with results averaged over three runs.
Results
Annealed CA-OPD beats OPD on every target metric: ScreenSpot-Pro is 45.92 versus 40.82, CC-OCR is 67.32 versus 63.93, and OCRBench-v2 Chinese is 53.22 versus 52.45. Fixed-threshold CA-OPD reaches 44.58 on ScreenSpot-Pro, while the reversed schedule reaches 44.36. Random writeback reaches only 40.20, and replacing CE with KL at repaired positions reaches 45.26, confirming separate benefits from placement, repair, and aligned supervision.
Applications
The method is directly relevant to mobile and desktop GUI agents that emit click coordinates, scrolling actions, and control selections. It also fits document OCR, receipt parsing, and multilingual screen reading. Prerequisites include a domain teacher, compatible tokenization, and structured outputs. Teacher computation is needed during training, but deployment can run only the compact student, preserving inference efficiency.
Limitations & Outlook
CA-OPD assumes that teacher probabilities are sufficiently calibrated and comparable across states. An overconfident teacher can write back incorrect tokens, while a weak teacher can systematically bias the student. Top-k KL may omit useful tail knowledge, and per-token teacher calls increase training cost. The current evidence is limited to image-based GUI and OCR tasks with large teachers; future work should study adaptive calibration, caching, video, and tool-use trajectories.
Plain Language Accessible to non-experts
Imagine training an apprentice to copy forms and operate a computer. At every step, the apprentice first proposes an answer. If the teacher sees that the character or click location is clearly unreliable, the teacher replaces it and makes the corrected answer the starting point for the next step. This prevents one early mistake from turning the rest of the form into nonsense.
The teacher does not take over forever. Early in training, supervision is strict because the apprentice is unreliable. As the apprentice improves, the teacher relaxes and intervenes only when a decision is genuinely risky. At corrected positions, the teacher gives the exact answer. At accepted positions, the teacher shows several plausible answers and how strongly each is supported, encouraging finer judgment.
That is CA-OPD: not simply more supervision, but supervision placed at the right moments and connected to the future context. ScreenSpot-Pro improves from 40.82 with ordinary OPD to 45.92, showing that targeted correction is more valuable than random assistance. The approach still costs more during training because the teacher must inspect proposals, but the trained student can operate alone.
ELI14 Explained like you're 14
Picture a video game where you must find a button, click it, and then type something. If your first click is wrong, the next screen may be completely different, so every later move becomes harder. Ordinary training is like making you play alone even after a bad opening. Another method gives you a perfect walkthrough, but that does not teach you what to do when you make your own mistakes.
CA-OPD is like having a smart coach. You make each move first. The coach checks how believable it is. If it looks really wrong, the coach replaces it and lets the next move start from the correct place. If it looks reasonable, you keep it, but the coach also explains which other moves might have worked. At the beginning the coach is strict; later, the coach lets you play more independently.
The researchers tested this on reading text and locating things on computer screens. A small Qwen3.5-0.8B model reached 45.92 on ScreenSpot-Pro, compared with 40.82 for ordinary OPD, and reached 52.47 on OCRBench-v2 English. Randomly helping did not work nearly as well, so the important idea is helping at the dangerous moments—not helping more often.
There are catches: asking the coach at every step makes training expensive, and the coach can also be wrong. The method has mainly been tested on GUI grounding and OCR. A next challenge is teaching video agents and tool-using assistants to recover from mistakes without needing a giant model all the time!
Glossary
On-Policy Distillation
A student generates its own training trajectories, and a teacher supervises the states the student actually visits. This reduces the mismatch between training prefixes and inference prefixes.
CA-OPD preserves student rollouts but selectively repairs unsafe transitions.
Negative Log-Likelihood (NLL)
NLL=-log p measures how much probability a model assigns to a candidate; lower values indicate stronger support. Unlike rank, it is an absolute confidence signal.
CA-OPD uses NLL to decide whether a student proposal is retained.
Prefix Repair
A teacher-corrected token is written into the autoregressive history, so future predictions condition on a safer prefix. It changes future states, not merely the current label.
The ablation shows that gating without writeback is almost as weak as OPD.
Forward KL
DKL(πT||πS) encourages the student to cover probability mass assigned by the teacher. CA-OPD approximates it using the teacher’s top-k predictions.
It supervises positions where the student proposal is retained.
Strict-to-Relaxed Schedule
The intervention threshold is strict early and relaxed later. This protects immature rollouts while progressively returning behavioral control to the student.
The paper implements the schedule with cosine threshold annealing.
Intervention-Aligned Supervision
The same gate determines both whether a token is written into the rollout and which loss is applied. Corrected positions use direct CE; retained positions use distributional KL.
This is the central coupling mechanism of CA-OPD.
Open Questions Unanswered questions from this research
- 1 Because gating depends on teacher probabilities, calibration may be decisive. Fixed NLL thresholds may not transfer across teachers, temperatures, domains, or token types.
- 2 The effect of top-k truncation on rare characters, long documents, and multilingual OCR remains unclear; tail probabilities may contain important structured information.
- 3 The trade-off among teacher calls, caching, training throughput, and intervention quality lacks a large-scale systems evaluation.
Applications
Immediate Applications
Reliable GUI agents
Mobile, desktop, and web automation teams can use a domain teacher to train a compact Qwen student that selectively repairs low-confidence coordinates and action tokens. Required assets include screenshots, instructions, structured actions, and a compatible teacher; deployment can run only the student.
Document and screen OCR
Receipt, form, and screen-reading systems can use confidence-aware correction when character or coordinate sequences become unreliable. The method is especially useful for teams that already possess domain OCR data and a larger teacher model.
Long-term Vision
Safe training for multimodal agents
The same principle could govern video navigation, tool calls, and long-horizon actions: retain teacher control before risky decisions, then release control as the student becomes reliable. Obstacles include calibration, latency, and scalable state verification.
Abstract
Autoregressive vision language models unify heterogeneous perception tasks but are highly susceptible to compounding errors. On-policy distillation (OPD) bridges the training-inference mismatch by training students on their own rollouts. However, unreliable student predictions, especially early in training, can derail the trajectory and degrade the quality of teacher supervision. While recent interleaved distillation methods allow the teacher to verify and replace student tokens, they primarily rely on rigid ranking metrics rather than exact teacher confidence, and they overlook how intervention decisions can inform token-level supervision. To address this, we introduce Confidence-Aware On-Policy Distillation (CA-OPD), a framework that couples reliable rollout construction with adaptive supervision. CA-OPD utilizes teacher confidence to selectively correct unreliable student transitions, gradually transferring rollout control to the student via a strict-to-relaxed schedule. Crucially, CA-OPD aligns knowledge transfer with these intervention decisions: corrected positions receive direct cross-entropy supervision from the teacher's prediction, while retained positions benefit from the teacher's full predictive distribution. Evaluated in a multi-teacher setting for GUI grounding and optical character recognition, CA-OPD substantially improves the Qwen3.5-0.8B baseline across all six target benchmarks, including gains of $9.50$ points on ScreenSpot-Pro and $6.72$ points on OCRBench-v2 English. Controlled studies further show that the gains depend on intervention placement, progressive rollout control, and intervention-aligned supervision, rather than intervention frequency alone.