The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
Introduces the 'Horizon Gap,' analyzing LLMs' planning, memory, and execution challenges in long-horizon tasks via 1,547-paper survey.
Key Findings
Methodology
A systematic literature review of 1,547 papers identifies six research directions: planning, memory, execution, training, evaluation, and foundational theory. Built using eight-thread seed harvesting and two-stage filtering.
Key Results
- Result 1: Outcome-only signals fail in long-horizon tasks; dense signals like process reward models and trajectory-level diagnostics are needed.
- Result 2: Disambiguates three axes: long-horizon (task property), long-context (model property), and long-term memory (system property).
- Result 3: Identifies two measurement challenges: decomposing model vs. harness capability and correlated bias in training/evaluation signals.
Significance
This work systematically defines and analyzes the primary bottleneck in long-horizon tasks, introducing the 'Horizon Gap' concept and providing a clear framework for future research.
Technical Contribution
Proposes a six-category taxonomy of task lifecycles, cross-analyzed with horizon-carrying mechanisms. Highlights diagnostic literature as a first-class thread throughout.
Novelty
First to systematically distinguish long-horizon, long-context, and long-term memory while introducing the Horizon Gap as a core bottleneck.
Limitations
- Limitation 1: Corpus construction relies on rule-based, single-annotator methods, risking classification bias.
- Limitation 2: Excludes non-English and unpublished industrial work, potentially underestimating progress.
- Limitation 3: Findings are based on literature review, lacking direct experimental validation.
Future Work
Future work could refine measurement methods, disentangle model vs. harness capabilities, and explore predictive theories for long-horizon reliability.
AI Executive Summary
While large language models (LLMs) excel at single-step reasoning tasks, they struggle with long-horizon tasks, such as multi-hour software engineering projects. Issues like goal drift, memory loss, and premature task completion highlight a gap termed the 'Horizon Gap.'
This paper systematically reviews 1,547 papers, identifying six key research areas: planning, memory, execution, training, evaluation, and foundational theory. It distinguishes three independent axes—long-horizon (task property), long-context (model property), and long-term memory (system property)—and finds that outcome-only signals degrade in long-horizon tasks, requiring dense signals like process reward models.
By introducing the Horizon Gap framework and outlining unresolved challenges, such as disentangling model and harness capabilities, this work provides a foundational roadmap for advancing LLMs in long-horizon tasks, bridging the gap between single-step benchmarks and real-world deployment.
Deep Analysis
Background
Recent advances in LLMs have enabled breakthroughs in single-step reasoning tasks like code generation. However, long-horizon tasks reveal limitations in planning, memory, and execution.
Core Problem
The core issue is the Horizon Gap: LLMs lose coherence in long-horizon tasks, leading to goal drift, forgotten decisions, and incomplete work.
Innovation
This paper introduces the Horizon Gap concept, systematically reviews 1,547 papers, and establishes a six-category taxonomy while distinguishing long-horizon, long-context, and long-term memory.
Methodology
- �� Corpus construction: Harvested 1,547 papers via eight-thread queries and two-stage filtering.
- �� Taxonomy: Categorized papers into six lifecycle stages, cross-analyzed with horizon-carrying mechanisms.
- �� Diagnostic focus: Treated critical literature as integral to all categories.
Experiments
The study analyzes process reward models, trajectory-level diagnostics, and long-horizon evaluation methods, confirming the Horizon Gap's impact on performance.
Results
Outcome-only signals degrade in long-horizon tasks; dense signals improve performance. Measurement challenges include disentangling model vs. harness capabilities and addressing correlated biases.
Applications
The findings support advancements in automated software development, complex task decomposition, and long-term user interaction systems.
Limitations & Outlook
Corpus construction and classification may introduce bias. Non-English and unpublished industrial work is excluded. Findings lack direct experimental validation.
Plain Language Accessible to non-experts
Imagine you're building a complex Lego castle. You need to plan each step, remember what you've built, and ensure all pieces fit together. LLMs are like a builder who forgets previous steps after each action. This study explores how to help the 'builder' remember and complete the entire castle.
ELI14 Explained like you're 14
Think of playing a long video game where you build a castle. You need to remember what you built earlier and plan your next steps. If you forget, your castle might fall apart! LLMs are like a forgetful gamer, and this research helps them stay on track and finish the game!
Glossary
Horizon Gap
The gap between single-step capability and reliable long-horizon task completion.
Central bottleneck identified in LLMs for long-horizon tasks.
Long-Horizon Task
Tasks requiring multiple sequential decisions to complete.
Focus of this paper's analysis.
Process Reward Model
Generates dense signals to replace outcome-only rewards.
Addresses signal degradation in long-horizon tasks.
Trajectory-Level Diagnostics
Evaluates task execution processes rather than just final outcomes.
Used to assess long-horizon task performance.
Long-Term Memory
System's ability to retain information across multiple steps or sessions.
Key to improving long-horizon task execution.
Open Questions Unanswered questions from this research
- 1 How to disentangle model vs. harness capabilities?
- 2 How to mitigate correlated biases in training and evaluation signals?
- 3 Does long-horizon reliability admit a general predictive theory?
Applications
Immediate Applications
Automated Code Generation
Enhances LLMs' planning and execution in long-term software development tasks.
Complex Task Decomposition
Supports breaking down multi-step tasks, such as project management.
Long-term Vision
Long-Term User Interaction
Enables intelligent assistants with persistent memory for personalized, cross-session services.
Abstract
Frontier language models solve reasoning problems in a single forward pass that would have been research contributions years ago, yet fail at multi-hour tasks: losing track of earlier decisions, declaring half-finished work done, or drifting from goals. We call this the horizon gap and survey 1,547 arXiv papers (2024-2026) collected via systematic seed harvest with a disclosed 26.8% bleed filter, extended by targeted supplementation. We disambiguate three routinely conflated properties: long-horizon (task property: required steps), long-context (model property: token capacity), and long-term memory (system property: persistence across steps/sessions). We organize the corpus into six categories tracking a long-horizon task's lifecycle -- planning, memory, execution, training, evaluation, and foundations/safety -- crossed with an axis capturing where horizons are carried (within-context, within-task-beyond-context, or cross-task-persistent). Across all categories, we find the same pattern: outcome-only signals grow uninformative as horizons lengthen, and the field's response -- whether process reward models, credit assignment, or trajectory-level diagnostics -- manufactures denser step-level signals. We treat critical and diagnostic literature as first-class threads throughout, arguing that segregating critique from method would routinely split single papers across chapters. We close by naming open measurement problems: decomposing model versus harness capability, managing correlated bias in process-level signals used for both training and evaluation, and whether long-horizon reliability admits general predictive theory.