Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs
GAMER constructs an Action-Centric Graph with dual-stream TD learning, boosting inference success by 20.81%.
Key Findings
Methodology
GAMER models historical reasoning as a dynamic directed graph where nodes represent discrete actions and edges encode temporal dependencies. It employs dual-stream TD learning to estimate positive (success likelihood) and negative (risk of failure) values for each action node, decoupling memory from LLM reasoning. During inference, high-value actions guide the search, while low-value actions prune the search space, reducing redundant exploration. The framework integrates these value signals into prompts, actively steering the LLM’s decision process, thus improving efficiency and resource utilization.
Key Results
- GAMER outperforms baselines with a 20.81% increase in success rate and 6.17% in progress rate across multiple benchmarks, including a 53.17% success boost on AlfWorld. It maintains stable performance across different LLMs (Qwen, Llama, GPT-4), with an average success rate of 40.30%. The method significantly reduces token consumption and search redundancy, demonstrating superior scalability and robustness.
- By leveraging dual-stream TD learning, GAMER effectively captures success and failure signals, enabling active pruning of invalid paths and guiding the search towards promising trajectories. This results in faster convergence and higher-quality solutions, especially in complex, multi-step tasks.
Significance
This work addresses the fundamental inefficiency of stateless inference in large language agents, introducing a structured memory mechanism that actively guides reasoning. It bridges the gap between inference-time scaling and episodic memory, enabling persistent, knowledge-driven decision-making. The approach enhances the scalability, efficiency, and robustness of AI agents, with broad implications for autonomous reasoning, planning, and long-horizon task solving in real-world applications.
Technical Contribution
The core innovation is the Action-Centric Graph combined with dual-stream TD learning, enabling dynamic, fine-grained value estimation for actions. This decouples memory from reasoning, allowing active search guidance without increasing token costs. Theoretical analysis proves the method’s stochastic dominance and efficiency gains, providing a solid foundation for scalable inference. The framework also introduces prompt-based integration of learned values, seamlessly guiding LLMs during inference.
Novelty
This is the first work to integrate a dynamic, learned action graph with dual-value estimation for inference scaling in interactive agents. Unlike prior static memory or retrieval-based methods, GAMER actively guides search via reinforcement learning, providing a structured, adaptable memory that evolves with experience. It fundamentally shifts the paradigm from stateless to memory-guided inference, offering a new avenue for scalable, efficient AI reasoning.
Limitations
- Dependence on quality and coverage of historical trajectories; poor or biased data can impair performance. Handling very long sequences or highly dynamic environments remains challenging, requiring further graph update mechanisms.
- Potential misjudgment of low-value actions in high-risk scenarios could lead to suboptimal pruning. Computational costs for graph maintenance and TD updates are non-trivial, especially in large-scale applications.
- Limited support for multi-modal data and real-time environment updates; future work should address these aspects for broader applicability.
Future Work
Future directions include integrating multi-modal data, enabling real-time graph updates, and extending to multi-agent systems. Combining meta-learning and transfer learning could improve adaptation to new tasks. Enhancing scalability for extremely large graphs and exploring unsupervised or semi-supervised learning for value estimation are also promising avenues.
AI Executive Summary
Recent advances in large language models (LLMs) have unlocked unprecedented reasoning capabilities, yet their inference mechanisms remain largely stateless, leading to redundant searches and high resource consumption. This inefficiency hampers their deployment in complex, long-horizon tasks requiring persistent memory and strategic planning. To address this, the paper introduces GAMER, a novel framework that constructs a dynamic Action-Centric Graph from historical trajectories, enabling active memory-guided inference.
GAMER employs dual-stream Temporal Difference (TD) learning to estimate positive (success likelihood) and negative (failure risk) values for each action node, effectively capturing nuanced knowledge about task progress and pitfalls. During inference, these learned values are integrated into prompts, guiding the LLM to prioritize high-value actions and avoid low-value ones, thus significantly reducing search space and computational costs.
Experimental results across multiple benchmarks, including AlfWorld, SciWorld, PDDL, and Tool, demonstrate that GAMER improves success rates by 20.81% and progress rates by 6.17% over baseline methods. The approach maintains robustness across different models such as Qwen, Llama, and GPT-4, confirming its generalizability. Theoretically, the framework guarantees stochastic dominance and efficiency gains, validating its scalability.
This work offers a transformative step toward persistent, knowledge-driven AI agents capable of efficient reasoning in dynamic environments. Future research will focus on multi-modal integration, real-time graph updates, and broader applications in autonomous systems, pushing the frontier of scalable, intelligent decision-making.
Deep Dive
Applications
What is the real-world impact?
Limitations & Outlook
What gaps remain?
Abstract
Recent advancements in inference-time scaling have significantly unlocked the complex reasoning capabilities of Large Language Models~(LLMs). However, for agents, these approaches suffer from a critical inefficiency, operating in a stateless manner and engaging in redundant search processes. Existing memory mechanisms largely rely on the reasoning capabilities of LLMs, leading to prohibitive computational costs. In this paper, we propose a novel framework, \textit{GAMER}~(Graph-based Action-centric Memory with Episodic Reasoning), that bridges the gap between inference scaling and episodic memory. Our approach models historical reasoning as a dynamic \textit{Action-Centric Graph}. By decoupling the memory mechanism from LLMs, our method can save token/money usage by providing less memory context than memory mechanism baselines. To extract knowledge from the graph effectively, we use a dual-stream Temporal Difference learning mechanism to estimate the positive~(suggestion) and negative~(avoidance) value of action nodes based on past successes and failures. During the inference phase, this learned value function optimizes decision-making bi-directionally, so that positive values provide action suggestions, while negative values indicate high-risk actions. By performing efficient searches on the graph, our method significantly improves the efficiency of inference scaling. Experiments on multiple benchmarks demonstrate that \textit{GAMER} achieves superior performance by \textbf{20.81\%/6.17\%} for success/progress rate compared to vanilla baselines.