DAPD: Dual-Anchored Policy Distillation
DAPD significantly improves privilege illusion by dual-path and dual-source anchoring, with an average gain of 2.00 points.
Key Findings
Methodology
DAPD is a unified framework addressing information asymmetry through Dual-Path Anchoring and Dual-Source Anchoring. Dual-Path Anchoring includes unconditioned and privileged paths to align behavior with and without privileged information. Dual-Source Anchoring applies these paths in both reference-to-rollout and rollout-to-reference directions, balancing reliability and reachability.
Key Results
- On Qwen3-4B, DAPD improves over OPSD by an average of 2.00 points across six tasks, notably achieving a 2.78-point gain at 32B scale, demonstrating stability across model scales.
- In out-of-distribution tests, DAPD gains 4.82 points on LCB v5, showing strong robustness.
- Ablation studies verify the information asymmetry mechanism, with complementary effects from dual-path and dual-source anchoring.
Significance
DAPD significantly alleviates privilege illusion, enhancing model accuracy and stability during inference. It offers a new solution in academia and provides a more reliable training method in industry, addressing long-standing information asymmetry issues.
Technical Contribution
By introducing self-conditioned distribution as a bridge, DAPD provides new theoretical guarantees and engineering possibilities, fundamentally differing from existing SOTA methods. It improves the use of privileged information and enhances inference capabilities through anchoring mechanisms.
Novelty
DAPD is the first to address privilege illusion through dual-path and dual-source anchoring, offering fundamental innovations in information alignment and guidance source balancing compared to existing methods.
Limitations
- In complex tasks, DAPD may still rely on privileged information, leading to performance drops.
- Additional computational resources are needed to handle anchoring mechanisms.
Future Work
Future research can explore DAPD's application in more domains and optimize its computational efficiency. Authors suggest further study on improving and extending anchoring mechanisms.
AI Executive Summary
Privilege illusion is a common issue in language model post-training, leading to poor inference performance. Existing methods fail to effectively address information asymmetry, causing student models to rely on privileged training information.
DAPD offers a solution through dual-path and dual-source anchoring. Dual-Path Anchoring aligns behavior with and without privileged information using a self-conditioned bridge, while Dual-Source Anchoring applies these paths in both reference-to-rollout and rollout-to-reference directions.
Experimental results show DAPD excels across multiple tasks and model scales, significantly improving inference accuracy. Despite limitations in complex tasks, its innovative anchoring mechanisms provide new directions for future research.
Deep Analysis
Background
Privilege illusion is a significant issue in language model post-training, leading to poor inference performance. Existing methods like OPSD fail to effectively address information asymmetry, causing student models to rely on privileged training information.
Core Problem
Privilege illusion causes student models to fail in reproducing privileged training information during inference, resulting in unsupported assertions and reasoning errors. This issue not only affects model performance but also limits its reliability in real-world applications.
Innovation
DAPD addresses information asymmetry through dual-path and dual-source anchoring. Dual-Path Anchoring aligns behavior with and without privileged information using a self-conditioned bridge, while Dual-Source Anchoring applies these paths in both reference-to-rollout and rollout-to-reference directions.
Methodology
- �� Dual-Path Anchoring: Aligns behavior with and without privileged information using a self-conditioned bridge.
- �� Dual-Source Anchoring: Applies paths in both reference-to-rollout and rollout-to-reference directions.
- �� Uses self-conditioned distribution as a bridge for information alignment.
Experiments
Experiments conducted on Qwen3 model series, covering reasoning, coding, and instruction-following tasks. Multiple benchmarks evaluate model performance, with ablation studies verifying anchoring mechanism effects.
Results
DAPD improves over OPSD by an average of 2.00 points on Qwen3-4B, notably achieving a 2.78-point gain at 32B scale. Ablation studies verify the information asymmetry mechanism, with complementary effects from dual-path and dual-source anchoring.
Applications
DAPD can enhance language model inference capabilities, suitable for scenarios requiring high accuracy and stability, such as automated customer service and intelligent assistants.
Limitations & Outlook
In complex tasks, DAPD may still rely on privileged information, leading to performance drops. Additional computational resources are needed to handle anchoring mechanisms.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef is preparing a complex dish. Existing methods are like having a detailed recipe during preparation but not during serving. DAPD is like providing the chef with a set of tools that can be used both during preparation and serving, ensuring the dish's quality and consistency.
ELI14 Explained like you're 14
Imagine you're playing a game where you have a super weapon during training but not in actual matches. DAPD is like giving you a tool that lets you use this weapon both in training and matches, ensuring you win the game!
Glossary
Privilege Illusion
Privilege illusion refers to the model's inability to reproduce privileged training information during inference, leading to performance drops.
Privilege illusion is the main issue in OPSD.
Dual-Path Anchoring
Dual-Path Anchoring aligns behavior with and without privileged information using a self-conditioned bridge.
A core mechanism of DAPD.
Dual-Source Anchoring
Dual-Source Anchoring applies paths in both reference-to-rollout and rollout-to-reference directions, balancing reliability and reachability.
Another core mechanism of DAPD.
Self-conditioned Distribution
Self-conditioned distribution refers to the model using the full completion information during prediction.
Used for information alignment in DAPD.
Information Asymmetry
Information asymmetry refers to the teacher model having privileged information that the student model lacks.
Root cause of privilege illusion.
Open Questions Unanswered questions from this research
- 1 How can DAPD be applied in more domains to enhance its generality and efficiency?
- 2 How can DAPD's computational resource requirements be optimized for better scalability?
Applications
Immediate Applications
Automated Customer Service
DAPD can improve the accuracy and stability of automated customer service, reducing incorrect responses.
Long-term Vision
Intelligent Assistants
DAPD can be used to develop smarter assistants, providing more reliable advice and guidance.
Abstract
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.