Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis
Trajectory-level framework analyzes 3,500 diagnostic paths, improving microservice root cause localization accuracy from 43.5% to 52.5% using fault propagation path validation.
Key Findings
Methodology
This study constructs a path-grounded evaluation framework by manually annotating service-level fault propagation paths as ground truth. It normalizes heterogeneous agent trajectories collected from six frameworks over 3,500 cases, employing metrics like Node F1 and Edge F1 to assess path reconstruction. The analysis investigates how agents explore the fault-impact surface, utilize evidence, and exhibit diagnostic intent. Failure modes are categorized into evidence omission, semantic misinterpretation, and unsupported inference. Based on these insights, a two-stage defense architecture, DiagGuard, is designed, combining evidence grounding and verification modules. Empirical results demonstrate that DiagGuard improves localization accuracy from 43.5% to 52.5% across different models and topologies.
Key Results
- Correctly localized faults show an average Node F1 of 0.75 and Edge F1 of 0.68, with deeper fault chains increasing reconstruction difficulty. Successful trajectories remain on the fault-impact surface, actively leveraging retrieved evidence and expanding query scope as search deepens. Failures often deviate from the true path, omit critical evidence, or misinterpret retrieved data. Implementing DiagGuard yields a 9% improvement in Acc@1, validating its effectiveness and generalization across models and network topologies.
- Path reconstruction metrics reveal that deeper causal chains pose significant challenges, and evidence utilization depth correlates strongly with diagnostic success. The framework exposes limitations of outcome-only metrics, emphasizing the importance of process-level analysis for trustworthy root cause diagnosis.
Significance
This work advances the field by shifting focus from final diagnosis correctness to process-level behavior analysis, revealing diagnostic biases and failure modes. It provides a granular understanding of how agents explore and utilize evidence, facilitating targeted improvements. The proposed framework enhances interpretability and trustworthiness of automated RCA, crucial for industrial deployment. Its insights into fault propagation path reconstruction lay a foundation for more reliable and explainable autonomous troubleshooting in complex microservice environments.
Technical Contribution
The paper introduces a novel trajectory-level evaluation framework that normalizes and compares heterogeneous diagnostic trajectories against annotated fault propagation paths. It develops metrics like Node F1 and Edge F1 for path reconstruction quality, and constructs an intent taxonomy based on SQL query analysis. The design of DiagGuard, a two-stage defense combining evidence grounding and verification, represents a significant step forward in improving root cause localization robustness. Empirical validation across multiple models and topologies demonstrates the framework’s broad applicability and effectiveness, marking a key technical contribution to AI-driven system diagnostics.
Novelty
This is the first comprehensive study integrating manual fault propagation path annotations with trajectory-level analysis of diagnostic agents. Unlike prior outcome-focused evaluations, it emphasizes process transparency and path reconstruction quality. The introduction of a two-stage defense mechanism, DiagGuard, grounded in evidence investigation and validation, offers a new paradigm for enhancing automated root cause analysis in microservices. Its focus on behavioral analysis and failure mode categorization sets it apart from existing static or purely statistical methods, pushing the frontier of explainable AI in system troubleshooting.
Limitations
- The reliance on manual annotation of fault propagation paths limits scalability and adaptability to dynamic or large-scale systems. Path reconstruction remains challenging in highly complex or rapidly changing environments. The models evaluated are static, and real-time, online fault diagnosis in production systems may face additional hurdles. Moreover, the approach assumes availability of detailed telemetry and logs, which may not always be feasible. Future work should explore automated path inference, real-time adaptation, and multi-modal data fusion to address these limitations.
Future Work
Future research will focus on automating fault propagation path annotation via learning-based inference, reducing manual effort. Enhancing real-time path reconstruction capabilities and integrating multi-source data, such as traces, logs, and metrics, will be prioritized. Additionally, extending the framework to handle dynamic, large-scale, and streaming environments is crucial. The authors also plan to develop adaptive query strategies and reinforcement learning-based optimization to further improve diagnostic efficiency and robustness, aiming for industrial deployment in complex, evolving microservice architectures.
AI Executive Summary
In modern microservice architectures, diagnosing the root cause of failures remains a complex challenge due to the intricate fault propagation paths and the dynamic nature of distributed systems. Traditional evaluation metrics, such as endpoint correctness, provide limited insight into the diagnostic process itself, often overlooking how agents explore evidence and reconstruct fault paths. This gap hampers the development of trustworthy automated troubleshooting tools.
To address this, the authors propose a trajectory-level evaluation framework that leverages manually annotated fault propagation paths as ground truth. By normalizing and analyzing 3,500 diagnostic trajectories collected from multiple agent frameworks, the study reveals critical behavioral patterns distinguishing successful from failed diagnoses. Successful agents tend to stay on the fault-impact surface, actively utilize retrieved evidence, and broaden their query scope as the search deepens. Failures often involve path deviations, evidence misinterpretation, or superficial exploration.
Building on these insights, the researchers introduce DiagGuard, a two-stage defense mechanism combining evidence grounding and verification modules. This architecture significantly improves root cause localization accuracy, raising the top-1 success rate from 43.5% to 52.5% across different models and network topologies. The results demonstrate that process-level trajectory analysis exposes hidden limitations of outcome-only metrics and provides actionable guidance for enhancing automated RCA systems.
Overall, this work advances the understanding of diagnostic behaviors in AI-powered microservice troubleshooting, emphasizing the importance of path reconstruction and evidence utilization. It offers a practical blueprint for designing more reliable, interpretable, and scalable fault localization tools, paving the way for more resilient cloud systems. Future directions include automating fault path inference, integrating multi-modal data, and deploying in real-time industrial environments, ultimately transforming how complex distributed systems are maintained and operated.
Deep Dive
Abstract
Existing evaluations of automated root cause analysis (RCA) for microservices assess diagnostic performance mainly by endpoint correctness: whether a method localizes the responsible service. This criterion enables comparison but does not reveal the evidentiary basis of a diagnosis or the fault-propagation route connecting the source to observed symptoms, both of which an on-call site reliability engineer needs to judge whether action is warranted. We therefore treat RCA as an observable diagnostic process. Our trajectory-level framework evaluates agent executions against manually curated service-level fault-propagation paths. Applied to a public microservice RCA benchmark, it analyzes 3,500 diagnostic trajectories, characterizing where agents investigate and how they use retrieved telemetry. We find a disconnect between answer correctness and diagnostic quality: an agent may localize the fault source yet fail to reconstruct its propagation. Successful investigations stay on the fault-impact surface, act on retrieved evidence, and broaden their query repertoire as the search deepens. Failures arise when decisive evidence is omitted, retrieved evidence is misinterpreted, or unsupported inference substitutes for missing evidence. We operationalize this taxonomy as DiagGuard, a two-stage defense-in-depth architecture in which grounding surveys available observations before localization and verification audits the diagnosis against them. In an independent setting with a different model, benchmark, and service topology, DiagGuard raises Acc@1 from 43.5% to 52.5%. These results show that trajectory-level evaluation exposes limitations hidden by final-answer metrics and provides actionable guidance for improving automated RCA.