Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agents
Agentic Monte Carlo (AMC) uses Bayesian posterior sampling with SMC to optimize black-box LLM agents, outperforming prompting and GRPO.
Key Findings
Methodology
AMC leverages the RL-Bayesian equivalence, defining the optimal policy as a posterior over trajectories. It employs a learned value function Vθ to guide sequential Monte Carlo (SMC) sampling, maintaining the black-box model unchanged. The process involves importance sampling and resampling to approximate the posterior, with Vθ trained via regression on trajectories. Experiments on AgentGym’s WebShop, SciWorld, and TextCraft environments show significant improvements over prompting baselines and outperform GRPO as test compute scales.
Key Results
- In WebShop with Llama-3.2-11B, AMC achieved a score of 0.625, surpassing prompt-based methods (0.562) and SMC (FoA) (0.580). On GPT-5.1, it improved from 0.519 to 0.543, demonstrating robustness across models.
- In SciWorld and TextCraft, AMC consistently outperformed baselines, with over 10% average gains, especially in sparse reward settings.
- Scaling test compute further enhanced AMC’s performance, surpassing gradient-based methods like GRPO, validating its effectiveness for black-box models.
Significance
This work addresses the critical challenge of optimizing proprietary, closed-source LLMs without access to parameters. By framing RL as Bayesian posterior sampling, it opens new avenues for reinforcement learning in industry settings, enabling autonomous agents to improve via principled probabilistic inference, thus broadening RL’s practical impact.
Technical Contribution
The paper introduces a novel approach translating RL into posterior sampling, training a value function Vθ to guide SMC sampling, and employing importance resampling to approximate the optimal policy. This circumvents the need for gradient access, providing a theoretically sound and practically feasible framework for black-box RL optimization.
Novelty
This is the first work to apply Bayesian posterior sampling via SMC for black-box LLM agents, integrating learned value functions to guide sampling without parameter access. It bridges the gap between RL and inference, offering a new paradigm for black-box model optimization, distinct from prior gradient-based or prompt-based methods.
Limitations
- The approach relies heavily on the accuracy of the learned value function; errors here can degrade sampling quality, especially in high-dimensional or noisy environments.
- Computational cost remains high due to multiple trajectory sampling and resampling steps, limiting real-time applications.
- In environments with extremely sparse rewards, the importance weights may become degenerate, requiring further algorithmic improvements.
Future Work
Future directions include developing more efficient value function training methods, reducing computational overhead, extending to continuous actions, and multi-agent systems. Exploring adaptive resampling strategies and integrating model compression techniques could further enhance scalability and robustness.
AI Executive Summary
The rapid advancement of large language models (LLMs) has transformed them from passive text generators into active autonomous agents capable of multi-step reasoning and environment interaction. However, the proprietary nature of state-of-the-art models like GPT-5 and Gemini 3 restricts access to model parameters, posing a significant barrier to applying traditional reinforcement learning (RL) techniques that rely on gradient information. Existing solutions mainly involve prompt engineering or fine-tuning open-weight models, which do not directly optimize the target black-box models.
This paper introduces Agentic Monte Carlo (AMC), a novel framework that leverages the theoretical equivalence between RL and Bayesian inference. Instead of updating model parameters, AMC formulates the optimal policy as a posterior distribution over trajectories, combining a fixed prior (the black-box model) with a likelihood encoding reward success. Using Sequential Monte Carlo (SMC), AMC samples from this posterior by learning a value function Vθ that guides the sampling process, effectively steering the agent toward optimal behaviors without modifying the underlying model.
The core technical innovation lies in training Vθ via regression on sampled trajectories, then employing importance resampling to approximate the posterior distribution. This approach allows for principled RL-style optimization on black-box models, circumventing the need for gradient access. Extensive experiments on AgentGym environments demonstrate that AMC outperforms prompt-based baselines and, as test compute increases, surpasses gradient-based methods like GRPO. These results highlight the potential of probabilistic inference techniques to unlock new capabilities in proprietary language models.
Overall, AMC offers a scalable, theoretically grounded method for optimizing black-box LLM agents, opening pathways for autonomous systems in industry and research. Future work will focus on improving sampling efficiency, extending to continuous actions, and applying to multi-agent scenarios, further broadening the impact of this innovative approach.
Deep Analysis
Background
近年来，预训练大规模语言模型（如GPT系列）在自然语言理解和生成中取得突破，逐渐演变为具备多步推理和环境交互能力的自主代理。传统RL方法（如PPO、GRPO）在开放参数模型上表现优异，但受限于模型参数的访问权限，难以应用于封闭式API模型。现有研究多依赖微调或提示工程，缺乏系统的RL优化框架。贝叶斯控制和推断方法为解决此类问题提供了理论基础，但实际应用仍受采样效率和模型限制影响。本文结合贝叶斯推断与SMC技术，旨在突破黑箱模型的优化瓶颈，为RL在私有模型上的应用提供新思路。
Core Problem
核心问题在于，黑箱LLM模型无法进行参数微调，传统RL算法依赖梯度信息，难以直接优化目标策略。如何在不访问模型参数的情况下，实现策略的优化和提升，成为关键难题。现有方法多局限于提示工程或微调开放模型，缺乏系统的无参数RL框架。本文试图通过贝叶斯后验采样，直接从轨迹分布中抽样最优策略，解决模型封闭带来的限制。这一问题的解决，将极大推动私有模型的自主学习和智能体发展。
Innovation
主要创新包括：1）将RL优化问题转化为贝叶斯后验采样，定义最优策略为轨迹的后验分布，避免参数微调；2）引入学习的价值函数Vθ，用于引导SMC采样，提升样本效率；3）采用序贯重要重采样（SIR）机制，有效逼近最优轨迹。此方法结合了贝叶斯推断的理论优势与SMC的高效采样机制，突破了模型参数不可访问的限制，提供了在黑箱模型上实现RL优化的全新路径。这一创新为未来黑箱模型的强化学习提供了理论支撑和实践工具。
Methodology
- �� 以黑箱LLM模型作为先验π，定义轨迹的后验分布，结合奖励信息形成目标。• 训练价值函数Vθ，通过回归未来奖励，提供轨迹优劣估计。• 利用SMC的序贯重要重采样，逐步采样轨迹，调整采样权重以逼近最优后验。• 在每个时间步，从π采样动作，利用Vθ计算重要性权重，进行重采样。• 通过学习的Vθ引导采样，避免模型参数微调，实现RL目标。• 最终得到一组轨迹，代表最优策略的近似样本。• 实验中在AgentGym环境验证，表现优于提示和基线方法。
Experiments
在AgentGym平台的WebShop、SciWorld、TextCraft环境中，使用不同模型（Llama-3.2-11B、GPT-4、GPT-5）验证AMC。对比提示基线、Best-of-N和SMC（FoA）方法，采用15轨迹采样，评估最高奖励轨迹。训练价值函数Vθ，采用Transformer架构，优化目标为未来奖励的回归。实验指标为环境奖励得分，AMC在所有环境中均优于对比方法，特别在稀疏奖励任务中表现出更强的稳定性。随着测试计算的扩大，AMC逐步超越参数微调的GRPO，验证其在黑箱模型上的优越性。
Results
AMC在WebShop中，Llama-3.2-11B模型上，最高奖励轨迹得分从提示的0.562提升到0.625，超越Best-of-15（0.562）和SMC（FoA）（0.580）。在GPT-5.1模型上，表现亦优于基线，提升约2个百分点。SciWorld和TextCraft中，AMC平均提升10%以上，尤其在稀疏奖励环境中表现出更强的稳定性。随着测试计算的扩大，AMC逐步超越依赖参数微调的GRPO，验证了其在黑箱模型上的RL优化潜力。
Applications
该方法适用于工业界的私有模型优化、自动化决策系统和自主智能体开发。无需模型参数访问，便于在封闭API环境中实现策略优化。未来可结合多智能体系统、连续动作空间，推动自主机器人、金融决策等领域的应用。
Limitations & Outlook
依赖价值函数的准确性，训练偏差可能影响采样效果。高维状态空间和复杂环境中，采样效率仍需提升。算法计算成本较高，尤其在大规模轨迹采样时，需优化效率。未来需解决稀疏奖励和样本退化问题，提升实用性。
Plain Language Accessible to non-experts
想象你在做一道复杂的菜，手边只有一个食谱（黑箱模型），你不能直接改动食谱，只能根据成品的味道不断调整。传统方法要你逐步试错，花费很多时间。现在，你用一种聪明的办法：先用一些试验（轨迹）来学习哪些调料能让菜更好吃（价值函数），然后用这个知识指导下一轮试验。通过不断筛选和调整，你最终找到最美味的做法。这就像用贝叶斯推断和蒙特卡洛采样，帮助你在不能直接改食谱的情况下，优化菜的味道。
ELI14 Explained like you're 14
假设你在玩一个游戏，但你不能直接修改游戏规则（模型参数），只能观察游戏的结果。你想找到赢的策略，但每次试错都很耗时间。于是，你用一种聪明的方法：先做一些尝试（轨迹），记录哪些尝试赢得多，然后用这些信息训练一个“聪明的小助手”——价值函数。这个小助手告诉你哪些动作可能赢得更多。接着，你用这个小助手指导下一轮尝试，优先选择可能获胜的动作。通过不断筛选和学习，你最终能找到最好的策略，即使不能直接改游戏规则。这就像用贝叶斯推断和蒙特卡洛采样，让你在看不见全部细节的情况下，也能找到最优方案。
Glossary
贝叶斯推断 (Bayesian Inference)
一种统计推断方法，通过结合先验信息和观察数据，计算后验概率。在论文中用以定义最优策略的后验分布。
定义最优策略为轨迹的后验分布，结合奖励信息进行采样。
序贯蒙特卡洛 (Sequential Monte Carlo)
一种重要性采样方法，用于估计动态系统的后验分布，逐步采样并重采样以逼近目标分布。
实现黑箱模型的轨迹采样与优化。
价值函数 (Value Function)
估算在某状态下未来奖励的函数，用于引导采样和策略优化。
Vθ训练用于引导SMC采样。
重要性重采样 (Importance Resampling)
根据样本的权重重新抽样，以逼近目标分布。
在SMC中逐步筛选最优轨迹。
黑箱模型 (Black-box Model)
模型参数不可访问，只能通过输入输出交互使用的模型。
如GPT-5、Gemini 3等API模型。
Open Questions Unanswered questions from this research
- 1 如何进一步提升价值函数Vθ在高维环境中的训练效率，尤其在奖励稀疏或噪声大的场景下，仍是挑战。
- 2 在多智能体或连续动作空间中，AMC的适应性和扩展性尚未充分验证，未来需探索更广泛的应用场景。
Applications
Immediate Applications
私有模型优化
企业可利用AMC在封闭API模型上实现策略优化，无需模型参数访问，提升自动化决策能力。
自主智能体开发
在机器人或虚拟助手中应用AMC，实现无需微调模型参数的自主学习与优化。
Long-term Vision
行业自主智能系统
未来AMC有望推动金融、医疗等行业的自主决策系统，降低对模型开放的依赖，提升安全性和隐私保护。
Abstract
LLM agents operate in two distinct regimes: open-weight agents amenable to reinforcement learning (RL) and black-box agents whose behaviour must be controlled purely at test time. Although black-box agents are often backed by state-of-the-art proprietary LLMs, API-only access precludes parameter-level optimization, rendering most RL methods inapplicable. To address this limitation, we turn to a known equivalence between RL and Bayesian inference. We propose Agentic Monte Carlo (AMC) to directly sample from the optimal policy of a black-box agent rather than training it through RL. The optimal policy is a posterior over trajectories whose prior we define as the fixed black-box LLM agent. We employ Sequential Monte Carlo to sample from this posterior by learning a value function to steer the agent while leaving the underlying black-box model unchanged. We validate AMC on three diverse environments from the AgentGym benchmark, demonstrating significant improvements over prompting baselines and even outperforming Group Relative Policy Optimization (GRPO) as we scale the test-time compute of our method. AMC demonstrates the feasibility of performing principled RL-style optimization of black-box LLM agents. Code is available at https://github.com/layer6ai-labs/Agentic-Monte-Carlo