Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agents
COTA enhances runtime intervention in LLM agents by comparing alternatives, improving performance in environments like WebShop.
Key Findings
Methodology
COTA is a comparison-only framework for constructive runtime intervention in LLM agents. It uses a tiny comparator to judge which of two actions from the same state leads to a better continuation. The comparator is trained using pairwise supervision from same-prefix counterfactual branches, offering non-binding advice and allowing the original actor to replan.
Key Results
- In WebShop, COTA improved Qwen3-8B's mean reward from 0.3960 to 0.5630, significantly outperforming baseline methods.
- In ALFWorld, COTA increased success rate from 82.84% to 90.30%, demonstrating strong intervention capability.
- In τ3-Retail, COTA raised success rate from 37.50% to 45.00%, significantly better than other methods.
Significance
COTA achieves effective runtime intervention without increasing task-solving capability by comparing alternatives. This method not only enhances agent reliability but also reduces dependency on additional task solvers, lowering computational costs.
Technical Contribution
COTA simplifies runtime intervention to a candidate-relative comparison problem, avoiding the need for task-solving capability in the auxiliary model. Its innovation lies in using a tiny comparator for local comparison while retaining task-level replanning capability.
Novelty
COTA is the first to simplify runtime intervention to a framework requiring only local comparison, avoiding the high task-solving demands of traditional methods and providing a more lightweight solution.
Limitations
- COTA may not provide sufficient guidance in complex tasks due to the limited capability of the comparator.
- Misjudgments may occur in extreme cases, leading to unnecessary interventions.
Future Work
Future research could explore enhancing the comparator's capability and validating COTA's effectiveness in more diverse environments.
AI Executive Summary
In modern AI, solving long-horizon tasks requires robust reasoning and decision-making capabilities. Traditional methods often rely on costly expert solvers or critics, increasing computational burden and potentially leading to over-intervention. To address this, researchers propose COTA, a comparison-only framework for constructive runtime intervention in LLM agents.
COTA's core is a tiny comparator that compares multiple actions from the same state to determine which leads to better outcomes. This approach allows COTA to provide effective intervention advice without increasing task-solving capability, significantly improving agent reliability and efficiency.
Experimental results show that COTA achieves significant performance improvements in environments like WebShop, ALFWorld, and τ3-Retail. This method not only reduces dependency on additional task solvers but also lowers computational costs, offering new directions for future AI research. Despite potential limitations in complex tasks, COTA's innovation and effectiveness provide important insights for the field's development.
Deep Analysis
Background
In recent years, LLM agents have been increasingly applied to long-horizon tasks, which typically require complex reasoning and decision-making capabilities. Traditional methods often rely on costly expert solvers or critics, increasing computational burden and potentially leading to over-intervention, affecting overall agent performance.
Core Problem
In long-horizon tasks, improving agent reliability and efficiency without increasing computational burden is a significant challenge. Existing methods typically rely on costly expert solvers or critics, increasing computational burden and potentially leading to over-intervention.
Innovation
COTA simplifies runtime intervention to a candidate-relative comparison problem, avoiding the need for task-solving capability in the auxiliary model. Its innovation lies in using a tiny comparator for local comparison while retaining task-level replanning capability.
Methodology
- �� COTA uses a tiny comparator for local comparison
- �� Comparator is trained using same-prefix counterfactual branches
- �� Provides non-binding advice, allowing the original actor to replan
- �� Validated in environments like WebShop
Experiments
Validated in environments like WebShop, ALFWorld, and τ3-Retail using various agents like Qwen3-8B. Experimental design includes baseline comparisons and different candidate action generation mechanisms.
Results
COTA improved Qwen3-8B's mean reward from 0.3960 to 0.5630 in WebShop; increased success rate from 82.84% to 90.30% in ALFWorld; raised success rate from 37.50% to 45.00% in τ3-Retail.
Applications
COTA can be applied to long-horizon tasks requiring high reliability and low computational cost, such as autonomous driving and intelligent customer service. Its lightweight design makes it easy to integrate into existing systems.
Limitations & Outlook
COTA may not provide sufficient guidance in complex tasks due to the limited capability of the comparator. Misjudgments may occur in extreme cases, leading to unnecessary interventions.
Plain Language Accessible to non-experts
Imagine you are in a large kitchen cooking a meal. You are the head chef, responsible for making the final dish decisions. COTA acts like a small assistant; it won't tell you exactly how to cook but will suggest different ingredient combinations before you decide. You can choose to follow its advice or stick to your plan. This assistant doesn't need to know how to make the whole dish but can help you avoid some potentially bad choices by comparing different ingredient combinations. This way, you can improve the success rate of your dishes without adding much extra work.
ELI14 Explained like you're 14
Imagine you're playing a game where you have to make a lot of choices. Every time you make a decision, the game gives you some feedback. COTA is like a smart buddy; it won't tell you exactly how to win the game but will suggest different choices before you decide. You can choose to follow its advice or continue with your plan. This way, you can improve your chances of winning the game without adding much extra work. Cool, right?
Glossary
LLM (Large Language Model)
A deep learning model capable of processing and generating natural language, typically with billions of parameters.
Used for reasoning and decision-making in long-horizon tasks.
COTA (Comparison-Only Tiny Advisor)
A framework for runtime intervention that provides advice by comparing alternatives.
Used in LLM agents to improve reliability.
Runtime Intervention
Intervention before an agent executes an action to improve task success rate.
Used to monitor and adjust agent decisions.
Counterfactual Branching
A method for training models by altering decision-point actions to observe different outcomes.
Used to train COTA's comparator.
Candidate-Relative Comparison
Judging which of multiple actions from the same state is better by comparison.
Core mechanism of COTA.
Open Questions Unanswered questions from this research
- 1 How to enhance COTA's guidance capability in more complex tasks, as the current comparator may not handle all scenarios.
- 2 How to reduce misjudgments in extreme environments to ensure intervention accuracy.
Applications
Immediate Applications
Intelligent Customer Service
Improve response accuracy and efficiency of customer service bots using COTA, reducing human intervention.
Long-term Vision
Autonomous Driving
Apply COTA in autonomous driving to enhance decision reliability and safety, reducing accident risks.
Abstract
LLM agents are emerging as an important paradigm for real-world tasks that require reasoning, tool use, and sequential decision-making. As these agents operate over longer horizons, runtime intervention offers a way to improve reliability without retraining the underlying actor. Failure detection alone is insufficient. Effective intervention must also provide a useful direction for recovery. Existing approaches often rely on an expert solver or a critic that generates task-specific corrections, incurring either the cost of another capable solver or the capacity demands of a task-capable critic. We introduce Comparison-Only Tiny Advisor (COTA), a comparison-only framework for constructive runtime intervention. In COTA, a tiny comparator judges whether sampled alternatives lead to better continuations than the actor's proposal, and repeated comparisons determine when intervention is warranted. We train the comparator using pairwise supervision constructed from same-prefix counterfactual branches. Preferred alternatives are returned as non-binding advice, leaving the original actor to replan. Across WebShop, ALFWorld, and tau^3-Retail with three actors, COTA improves all nine evaluation settings and outperforms the compared baselines. These results show that constructive runtime intervention can remain effective even when the auxiliary model has substantially weaker task-solving capability than the actor.