Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

Recent advances in large language models (LLMs) have enabled their deployment as autonomous agents capable of tool use, dialogue, and web interactions. Early training relied on behavior cloning (behavior imitation) using expert trajectories, which, while simple, often failed to produce robust decision-making in complex environments. To improve, methods like reinforcement learning with human feedback (RLHF) and preference learning emerged, but these require environment interaction and reward models, increasing costs. Recent works have explored trajectory structuring, local error correction, and self-play, yet these approaches still depend on costly online interactions or human annotations. The challenge remains: how to leverage expert data efficiently without environment calls, especially for low-resource settings.

The core issue is that behavior cloning only captures sequence imitation, lacking the ability to correct mistakes or adapt to new situations. Existing preference and reinforcement learning methods improve decision quality but are resource-intensive, requiring environment rollouts, reward models, and full trajectory exploration. This limits scalability, especially for large models and complex tasks. The key question is how to utilize expert trajectories offline, transforming them into actionable signals that guide policy improvement without environment interaction. Addressing this gap is crucial for deploying scalable, cost-effective AI agents capable of robust decision-making in real-world scenarios.

The paper introduces several innovations: 1) Converting expert trajectories into state-conditioned preferences using a contrastive loss, 2) Sampling negative actions from the current model to provide local decision-level feedback, 3) Employing schema-based multi-view rendering (PPA) to stabilize preference signals across surface variations, 4) Incorporating SFT anchors to maintain format stability, 5) Achieving offline policy optimization without environment calls or reward models. These innovations collectively enable low-cost, high-performance agent training, addressing the limitations of prior methods that rely heavily on environment interactions or trajectory-level signals.

• �� Extract expert trajectories comprising states and latent actions.
• �� At each expert state, sample a schema view and render the current state.
• �� Generate K candidate actions from the current model, parse them into latent actions, and select the most probable negative sample different from the expert.
• �� Construct preference triples (state, expert action, negative action) under multiple schemas.
• �� Use a DPO-style loss to optimize the model, encouraging preference for expert actions over negatives.
• �� Incorporate SFT anchors to stabilize format and schema consistency.
• �� Apply PPA to render multiple schema views, reducing schema bias.
• �� Repeat over multiple rounds, updating the policy offline, without environment interaction.

The authors evaluate on three benchmarks: StableToolBench for tool use, τ-bench retail for long-horizon interactions, and Mind2Web for web GUI tasks. They compare against SFT, PPA+SFT, DPO variants, and online RL (GRPO). Metrics include accuracy, success rate, and step success. Models include Qwen3.5-2B/4B/9B and Gemma3-4B. Hyperparameters such as negative samples (K=4), preference coefficient (β), and refresh rounds (R=5) are tuned. Ablation studies assess the impact of each component, confirming the importance of SFT anchors and PPA. Results demonstrate consistent performance gains across scales and tasks, with statistical significance.

Agentic-DPO achieves a 21.7% to 41.4% accuracy increase on StableToolBench for 9B models, surpassing online methods with fewer resources. On τ-bench retail, success rates double, reaching 41.4%. In Mind2Web, step success improves to 64.4%. Ablation studies show removing SFT anchors or PPA causes performance drops, validating their stabilizing role. The method maintains robustness under data scarcity and perturbations, indicating strong generalization. The results collectively demonstrate that expert trajectories, when processed via preference contrast and schema augmentation, can effectively guide offline policy learning.

This approach is immediately applicable to scenarios requiring low-cost, offline policy training, such as customer service bots, web automation, and interactive assistants. It only needs expert demonstrations, avoiding environment calls or reward models, thus reducing deployment costs. Long-term, integrating multi-modal data and multi-task learning could enable autonomous agents capable of complex reasoning and adaptation across domains, transforming industries like healthcare, finance, and manufacturing where data efficiency and robustness are critical.

Dependence on high-quality expert data limits applicability in domains with sparse or biased demonstrations. Schema rendering and PPA may require task-specific tuning, affecting generalization. The method's stability and hyperparameter sensitivity need further investigation. Additionally, while avoiding environment interaction reduces costs, training still demands significant computational resources, especially for larger models. Future work should focus on broadening task scope, improving schema generalization, and integrating online adaptation to address these limitations.