PANDO: Efficient Multimodal AI Agents via Online Skill Distillation
PANDO improves multimodal AI agent efficiency via online skill distillation, achieving 58.3% success rate.
Key Findings
Methodology
PANDO is an online skill distillation framework featuring a structured Skill Library, progress reflection, confidence-based skill demotion, hierarchical routing, visual compression, and cache-aware prompting. Each component plays a role in reducing repeat-action loops, hidden discovery costs, and increasing prompt-cache reuse.
Key Results
- PANDO achieves a 58.3% success rate on 910 VisualWebArena tasks, outperforming SGV's 54.0% and WALT's 45.2%, while using 58% fewer tokens than SGV and 61% fewer than WALT.
- A 300-task ablation shows rules and routines provide most success gains, while routing, compression, and cache-aware prompting convert the larger skill library into lower marginal token cost.
- Three trajectory-level efficiency metrics—Action Repetition Rate, Step Overhead Ratio, and Prompt Cache Utilization—are introduced to make efficiency visible beyond terminal success.
Significance
This research addresses the high computational cost of inference in multimodal web agents, making agents more efficient as they accumulate experience. It offers academia and industry a new approach to enhance AI agent efficiency, reducing computational resource consumption.
Technical Contribution
PANDO innovates on existing methods by providing an online skill distillation approach that induces skills during the test-query stream without a pre-evaluation discovery budget. It demonstrates how to lower inference costs through a structured skill library and cache-aware prompting.
Novelty
PANDO is the first to propose an online skill distillation framework, differing from existing skill discovery methods by eliminating the need for offline tool discovery budgets and enhancing efficiency through real-time skill induction.
Limitations
- PANDO may perform poorly in dynamic task distributions as it assumes a fixed task distribution.
- The skill demotion mechanism might erroneously remove useful skills.
Future Work
Future research could explore optimizing PANDO's performance in dynamic task environments and improve the skill demotion mechanism to reduce the risk of mistakenly removing useful skills.
AI Executive Summary
Recent advances in multimodal web agents have largely relied on increased inference-time computation, including rollout search, verifier passes, offline skill discovery, and specialist model stacks. This raises a central question: can a web agent become more efficient as it accumulates experience, rather than more expensive?
PANDO is an online skill distillation framework designed to address inefficiencies such as repeat-action loops, hidden discovery costs, and low prompt-cache reuse in VisualWebArena. It enhances efficiency by maintaining a structured Skill Library and combining progress reflection, confidence-based skill demotion, hierarchical routing, visual compression, and cache-aware prompting.
On the full set of 910 VisualWebArena tasks, PANDO achieves a 58.3% success rate, outperforming SGV's 54.0% and WALT's 45.2%, while using 58% fewer tokens than SGV and 61% fewer than WALT. This demonstrates PANDO's ability to maintain high success rates while improving efficiency.
Deep Dive
Abstract
Recent advances in multimodal web agents often rely on increased inference-time computation, including rollout search, verifier passes, offline skill discovery, and specialist model stacks. This raises a central question: can a web agent become more efficient as it accumulates experience, rather than more expensive? We first analyze trajectories from VisualWebArena and identify three recurring sources of inefficiency: repeat-action loops, hidden discovery costs, and low prompt-cache reuse. We then introduce PANDO, a single-rollout online skill-distillation framework that maintains a structured Skill Library and combines progress reflection, confidence-based skill demotion, hierarchical routing, visual compression, and cache-aware prompting. On the full set of 910 VisualWebArena tasks, PANDO achieves a 58.3% success rate, outperforming SGV (54.0%) and our WALT reproduction (45.2%), while using 58% fewer tokens than SGV and 61% fewer tokens than WALT, without any pre-evaluation discovery budget. A 300-task ablation further shows that rules and routines provide most of the success gains, while routing, compression, and cache-aware prompting convert the larger skill library into lower marginal token cost. Finally, we introduce three trajectory-level efficiency metrics -- Action Repetition Rate, Step Overhead Ratio, and Prompt Cache Utilization -- to make efficiency visible beyond terminal success.