The Importance of Directional Feedback for LLM-based Optimizers
OptAgent uses directional feedback for text optimization; across four functions and 10-step trials, it is more stable and efficient.
Key Findings
Methodology
The paper casts LLM optimization as iterative maximization and classifies feedback as directional, non-directional, or reward-only. Sequential Prompt Optimization combines an Agent, Prompt Proposal module, Feedback Synthesizer, and Prompt Selector. From historical (o,r,f,p) tuples, it proposes a new prompt and accepts it only when its expected reward is no lower than the incumbent.
Key Results
- On Booth, McCormick, Rosenbrock, and Six-Hump Camel, every condition used 10 trials and at most 10 steps. For both GPT-3.5 and GPT-4, directional feedback substantially reduced cumulative and simple regret relative to no feedback and approached fixed-small-learning-rate SGD.
- GPT-4 could exploit non-directional feedback even when it only identified the relevant coordinate. However, directional feedback was more informative because it specified whether the coordinate should increase or decrease. Synthetic feedback was weaker than environmental feedback but consistently better than no feedback.
- In poem tasks requiring 7, 8, 9, or 10 syllables per line, OptAgent used line-level increase/decrease instructions to improve prompts over 10 rounds. It produced more reliable progress than or comparable advantages over the Reflexion baseline.
Significance
The study shows that LLMs are not limited to random prompt rewriting: feedback that specifies a direction enables behavior analogous to gradient search. This offers a common lens for prompt engineering, code debugging, agent planning, and human–AI interaction, and explains why reward-only black-box optimization is often unstable.
Technical Contribution
The work formalizes natural-language feedback as first-order-like information in text space and separates proposal from acceptance. Its Feedback Synthesizer asks how input changes could produce larger output changes, unlike APO's error-explanation framing. The Prompt Selector enforces an expected-reward non-decrease criterion, creating an explicit, modular improvement loop.
Novelty
The main novelty is not merely using LLMs to rewrite prompts, but isolating feedback directionality as a causal factor in optimization quality. Unlike APO, OPRO, and Reflexion, the paper evaluates the same principle on controlled mathematical landscapes and constrained poem generation, comparing direct, synthetic, and absent directional information.
Limitations
- The benchmark is small: four two-dimensional functions, 10 trials, and a 10-step horizon cannot establish performance in high-dimensional, noisy, or realistic nonconvex problems.
- The poem environment relies on programmable syllable checking. Open-ended quality, semantics, style, and conflicting human preferences are much harder to translate into reliable directions.
Future Work
Future studies should cover high-dimensional continuous variables, discrete code, and long-horizon agents; model uncertainty, feedback cost, and adaptive step sizes; compare more models and temperatures; and build benchmarks with authentic human feedback.
AI Executive Summary
LLMs are increasingly used to generate prompts, code, and plans, yet treating them as optimizers often produces random edits, repeated trial-and-error, and unstable rewards. Classical optimization draws a sharp distinction between black-box search and methods with a search direction: a gradient says where parameters should move. Nie and colleagues ask whether the same distinction governs optimization in language.
They introduce Sequential Prompt Optimization and instantiate it as OptAgent. The system records each prompt, output, reward, and feedback; a Prompt Proposal module generates a new prompt, a Feedback Synthesizer can convert scalar rewards into advice about how the input should change, and a Prompt Selector retains the candidate only when expected reward does not decrease. The design separates directional information, text editing, and acceptance decisions.
Across Booth, McCormick, Rosenbrock, and Six-Hump Camel, GPT-3.5 and GPT-4 ran up to 10 steps for 10 trials. Directional feedback lowered both cumulative and final regret, with performance approaching SGD; it also helped the weaker model. Synthetic feedback was inferior to direct feedback but better than none. In poem tasks requiring 7–10 syllables per line, OptAgent improved prompts over 10 rounds and outperformed or stabilized performance relative to Reflexion. The central lesson is that optimization depends not only on model intelligence but on whether feedback encodes an actionable direction. The evidence remains limited by small benchmarks, short horizons, and programmatic evaluators.
Deep Analysis
Background
Recent methods such as APO and OPRO mainly optimize prompts that improve another LLM, while Reflexion uses self-critique and interaction history. Classical optimization distinguishes gradient-based updates from evolutionary or Bayesian black-box search. This paper transfers that distinction to language, asking whether natural-language feedback can function as first-order information.
Core Problem
The objective is to find ptunable maximizing E[r|π(ptask,p)]. Text has no canonical algebraic difference, so an optimizer must infer both what to change and how much. With only scalar rewards, it is unclear whether an LLM can infer a useful search direction from past prompts and outcomes.
Innovation
The paper contributes four linked ideas: a taxonomy of directional, non-directional, and reward-only feedback; Sequential Prompt Optimization with modular proposal and selection; Feedback Synthesizer for recovering direction from numerical observations; and a cross-domain evaluation connecting controlled numerical optimization to constrained poem prompt optimization.
Methodology
- �� The Agent maps ptask and ptunable to output o, then receives reward r and feedback f; it does not independently see history.
- �� Prompt Proposal Δ reads task context, historical prompts, rewards, and feedback, and may rewrite the prompt completely.
- �� Feedback Synthesizer ˆF generates advice about input changes when the environment supplies only (o,r).
- �� Prompt Selector compares incumbent and candidate expected rewards and keeps the candidate only if it is no worse.
- �� Numerical tasks use R(x)=-J(x) and directional feedback ∇R(x); poem tasks report per-line syllable counts and required increase/decrease directions.
Experiments
The numerical benchmark contains Booth, McCormick, Rosenbrock, and Six-Hump Camel functions with two-dimensional x. Comparisons include GPT-3.5, GPT-4, SGD, no feedback, non-directional feedback, and directional feedback. Each condition uses 10 trials and at most 10 steps; metrics are simple regret |J(xT)-J(x*)| and cumulative regret. Poem tasks require 7, 8, 9, or 10 syllables per line, use full Algorithm 1, and compare OptAgent with Reflexion.
Results
Figure 2 shows that GPT-4 can infer a rough direction from history without feedback, whereas GPT-3.5 often fails; both models improve with explicit direction. Figure 3 reports final regret close to SGD under directional feedback. GPT-4 is not always closer at the final step, plausibly because models choose their own step sizes within only 10 steps. Figure 4 shows synthetic feedback below direct feedback but above no feedback. Figure 5 shows round-by-round reward gains for OptAgent across all syllable tasks.
Applications
The framework can support automatic prompt engineering, code debugging, API parameter tuning, planning policies, and constrained content generation. It is most useful when an evaluator can provide rewards and, ideally, indicate which attribute should increase or decrease; with score-only environments, a synthesizer can approximate this signal.
Limitations & Outlook
The paper reports no large cross-model statistical study or universal percentage improvement. Tasks are low-dimensional and short-horizon, and candidate selection requires additional evaluations. Directional feedback is difficult to obtain for open-ended writing, while LLM-chosen step sizes may overshoot or settle in local regions. Future work should address high-dimensional discrete spaces, noisy multi-objective rewards, and human evaluation.
Plain Language Accessible to non-experts
Imagine a restaurant kitchen trying to improve a recipe. After each dish, a customer gives a score. A score of 6 tells the chef that the result is mediocre, but not whether to add salt, reduce sugar, or change the cooking time. That is reward-only feedback. A comment such as “too salty—use less salt” is directional feedback: it identifies the way to improve without specifying the exact number of grams.
OptAgent acts like a head chef. It stores each recipe, dish, score, and comment, then writes the next recipe. After testing it, the system compares the new dish with the previous one; if it is not better, it keeps the old recipe. The process is therefore guided experimentation rather than random restarting.
The mathematical experiments resemble walking through a valley toward the lowest point. Directional feedback is someone pointing toward the better path; no feedback is wandering while checking only the altitude. In the poem task, the system is told that a line needs more or fewer syllables and gradually edits the writing instruction. The paper’s message is simple: results matter, but actionable advice about the direction of change makes improvement much more reliable.
ELI14 Explained like you're 14
Picture changing your character’s gear in a video game to get a higher score. After every attempt, the game might say only “score: 42,” or it might say “your character is too slow—raise agility.” Which message helps more? Obviously the second one! It tells you which way to move, even though it does not tell you the perfect number of points.
That is what this paper tests with GPT-3.5 and GPT-4. The models optimize math problems and also rewrite instructions for another model that writes poems. The poem must have a specific number of syllables per line. After each round, the optimizer suggests a new instruction and checks whether the result improved. If it gets worse, the old instruction stays.
The result is pretty clear: when the model hears “increase” or “decrease,” it searches much better. With only scores, GPT-4 can sometimes guess a pattern from earlier attempts, but GPT-3.5 wanders more. When direct advice is missing, another LLM can summarize what should change, and that still helps.
So LLM optimization is a bit like improving a gaming strategy. Knowing whether you won is useful, but knowing which move to change is much better! The catch is that math functions and syllable counting are neat and measurable. Real writing and real people are messier, so harder tests are still needed.
Glossary
Directional Feedback
Feedback that indicates the direction in which an input should change, without necessarily specifying the exact step size. It is the natural-language analogue of first-order information.
The numerical experiment uses ∇R(x), while the poem environment says whether each line needs more or fewer syllables.
Non-directional Feedback
Feedback that identifies a relevant attribute or violation but does not say whether it should increase or decrease.
The experiments identify which coordinate or poem lines need attention without providing a sign.
Sequential Prompt Optimization
An iterative framework that stores prompts, outputs, rewards, and feedback, then proposes, evaluates, and selects new prompts.
It is formalized as Algorithm 1 and instantiated by OptAgent.
Feedback Synthesizer
A module that turns historical inputs and numerical rewards into natural-language advice about useful changes.
It supplies approximate directional information when the environment returns scores only.
Simple/Cumulative Regret
Simple regret measures the final distance from the optimum; cumulative regret sums distance over all steps. They measure final accuracy and optimization efficiency.
Both metrics evaluate the four mathematical functions.
Open Questions Unanswered questions from this research
- 1 Does directional feedback remain reliable in high-dimensional, discrete, noisy text spaces? The paper studies only two-dimensional functions and short prompt trajectories.
- 2 Human feedback is ambiguous, costly, and often multi-objective. The field still needs calibrated synthetic feedback, preference-conflict handling, and realistic user benchmarks.
Applications
Immediate Applications
Automated prompt engineering
Teams can log prompts, outputs, and evaluation scores, then ask Feedback Synthesizer for actionable edits. Prompt Selector can retain improved versions using offline evaluation, supporting classification, summarization, and constrained generation.
Code and parameter debugging
An executor can return failed tests, error locations, and performance scores. The optimizer translates them into changes such as adding boundary checks, reducing latency, or tuning API parameters, then validates candidates with regression tests.
Long-term Vision
Auditable autonomous agents
Future agents could unify user comments, environment signals, and interaction histories into explicit directional advice for planning, robotics, and long-horizon tasks, while retaining prior policies as safe fallbacks.
Abstract
We study the potential of using large language models (LLMs) as an interactive optimizer for solving maximization problems in a text space using natural language and numerical feedback. Inspired by the classical optimization literature, we classify the natural language feedback into directional and non-directional, where the former is a generalization of the first-order feedback to the natural language space. We find that LLMs are especially capable of optimization when they are provided with {directional feedback}. Based on this insight, we design a new LLM-based optimizer that synthesizes directional feedback from the historical optimization trace to achieve reliable improvement over iterations. Empirically, we show our LLM-based optimizer is more stable and efficient in solving optimization problems, from maximizing mathematical functions to optimizing prompts for writing poems, compared with existing techniques.