TRAIL: Trace Reasoning and Agentic Issue Localization
TRAIL framework evaluates multi-agent systems using 148 human-annotated traces; best model achieves only 11%.
Key Findings
Methodology
Introduced a refined error taxonomy covering reasoning, planning, and execution. Built the TRAIL dataset with 148 traces grounded in GAIA and SWE-Bench benchmarks.
Key Results
- Result 1: GEMINI-2.5-PRO achieved only 11% joint accuracy in error categorization and localization tasks, highlighting LLM limitations in complex trace analysis.
- Result 2: Models frequently exceeded context length limits, requiring near-maximal input and output capabilities.
- Result 3: Reasoning chain augmentation significantly improved performance, underscoring the importance of exploration capabilities.
Significance
Addresses the critical challenge of evaluating complex agentic traces systematically, providing tools for debugging and optimization in academia and industry.
Technical Contribution
Developed a granular error taxonomy spanning reasoning, planning, and execution; constructed ecologically valid TRAIL dataset; revealed limitations of current LLMs in long-context debugging.
Novelty
First to propose a real-world trace-based error taxonomy and validate its effectiveness in debugging multi-agent systems.
Limitations
- Limitation 1: Dataset focuses on software engineering and information retrieval tasks, limiting domain coverage.
- Limitation 2: Model performance constrained by context length, unable to handle ultra-long traces.
- Limitation 3: Experiments lack detailed exploration of multi-agent collaboration failure modes.
Future Work
Future work could extend to diverse domains, enhance models' long-context handling, and explore multi-agent collaboration error patterns.
AI Executive Summary
The widespread adoption of agentic systems across domains has created an urgent need for scalable evaluation methods for their complex trace outputs. Existing approaches rely on manual human analysis, which cannot keep pace with increasing complexity and data volume. This study introduces the TRAIL framework, featuring a refined error taxonomy and a dataset of 148 real-world traces covering reasoning, planning, and execution failures. Experimental results reveal that even the best-performing model, GEMINI-2.5-PRO, achieves only 11% joint accuracy in error categorization and localization tasks, exposing limitations in long-context debugging.
The TRAIL dataset is grounded in GAIA and SWE-Bench benchmarks, focusing on software engineering and information retrieval tasks. Findings highlight the significant performance boost from reasoning chain augmentation, emphasizing exploration capabilities. However, challenges remain in handling ultra-long contexts and multi-agent collaboration errors.
TRAIL provides systematic tools for debugging and optimizing agentic workflows, with broad implications for academia and industry. Future work should expand domain coverage, improve long-context processing, and investigate multi-agent collaboration failure patterns.
Deep Dive
Abstract
The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.