ESPO: Error-Structured Prompt Optimization via Diagnose, Diversify, and Stabilize
ESPO enhances prompt accuracy by 3.76% and shortens prompts by 47% through structured error diagnosis, multi-strategy candidate generation, and bootstrap stability selection.
Key Findings
Methodology
ESPO decomposes prompt optimization into three phases: Diagnose, Propose, and Select. It uses large language models to cluster training errors into 3-7 patterns, ensuring comprehensive error coverage in a single round. Candidate prompts are generated via four strategies—diagnostic revision, consolidation, ablation, and factual injection—each addressing different error types with independent biases. The selection phase employs bootstrap resampling to identify the most robust prompt across multiple validation splits, grounded in theoretical generalization bounds. This approach transforms evolutionary search into a principled statistical estimation process, reducing redundancy and overfitting.
Key Results
- On seven public NLP benchmarks (Tweet, MMLU, GSM8K, HotpotQA, ScoNe, HoVer, PUPA), ESPO improves average accuracy by 3.76 percentage points (74.67% vs. 70.91%), while producing prompts 47% shorter (1004 vs. 1878 characters). It also achieves faster inference times. For example, on Qwen3 GSM8K, accuracy jumps from 15% to 91.4%.
- Cross-model experiments with Gemma 3 12B, Mistral 14B, Qwen3 32B, and Claude Haiku 4.5 show consistent improvements, with the largest gap on Qwen3 GSM8K (+76.4%). This demonstrates strong transferability of the method across different model sizes.
- Ablation studies confirm that adding diversity without bootstrap selection degrades performance by 1.2%, highlighting the importance of stable candidate selection. The structured error diagnosis and bootstrap filtering are key to the gains.
Significance
This work addresses fundamental limitations of evolutionary prompt optimization—prompt bloat, limited search diversity, and unreliable selection—by introducing a structured, statistically grounded framework. It advances both theoretical understanding and practical performance, enabling more concise, accurate, and efficient prompts. The approach enhances the robustness and generalization of prompt tuning, facilitating deployment in real-world NLP applications such as question answering, instruction following, and multi-task learning. Its theoretical guarantees and empirical results mark a significant step toward automated, scalable prompt engineering.
Technical Contribution
ESPO's core innovation lies in replacing heuristic evolutionary search with a three-phase statistical estimation framework. It employs large language models for error clustering, ensuring comprehensive error coverage in a single round. Multiple candidate generation strategies leverage error diagnosis to produce diverse prompts without bloat. The bootstrap-based selection guarantees robustness against validation noise, grounded in generalization bounds derived from PAC-Bayes and MDL principles. This unified approach surpasses existing methods like GEPA by producing shorter prompts with higher accuracy and better transferability, supported by rigorous theoretical guarantees.
Novelty
This is the first work to integrate structured error diagnosis with multi-strategy candidate generation and bootstrap stability selection in prompt optimization. Unlike prior methods relying solely on evolutionary mutations, ESPO systematically diagnoses error patterns, generating concise, targeted prompts. Its theoretical grounding in generalization bounds and the transformation of evolutionary search into a statistical estimation process represent a novel paradigm shift, offering both practical improvements and foundational insights.
Limitations
- Despite its effectiveness, ESPO's computational cost remains high due to multiple model inferences during candidate generation and bootstrap evaluation, limiting scalability in resource-constrained settings.
- The accuracy of error clustering depends on the diagnostic model's ability to identify meaningful error patterns; complex or subtle errors may be missed, reducing effectiveness.
- The assumption of independence among error clusters simplifies theoretical analysis but may not hold in practice, potentially affecting the robustness guarantees.
Future Work
Future research will focus on reducing computational overhead, possibly via model distillation or approximation techniques. Enhancing error clustering accuracy for more subtle errors, integrating multi-model diagnostics, and extending the framework to multi-modal tasks are promising directions. Additionally, combining ESPO with reinforcement learning to adaptively refine candidate strategies could further improve performance and applicability in industrial NLP systems.
AI Executive Summary
Prompt engineering has become a critical component in harnessing the full potential of large language models (LLMs). Traditional manual tuning is labor-intensive and lacks scalability, prompting the development of automated prompt optimization algorithms. Among these, evolutionary algorithms like GEPA have shown promise by iteratively mutating and selecting prompts based on model feedback. However, these methods suffer from prompt bloat, limited search diversity, and unreliable candidate selection, which hinder their efficiency and generalization.
This paper introduces ESPO (Error-Structured Prompt Optimization), a novel framework that redefines prompt tuning as a structured statistical estimation problem. ESPO decomposes the process into three phases: structured error diagnosis, multi-strategy candidate generation, and bootstrap-based stable selection. In the diagnosis phase, large models cluster training errors into a small set of root causes, ensuring comprehensive error coverage in a single round. The candidate generation phase employs four complementary strategies—diagnostic revision, consolidation, ablation, and factual injection—to produce diverse, concise prompts tailored to different error types. The final selection phase uses bootstrap resampling to identify prompts that consistently perform well across multiple validation splits, providing theoretical guarantees grounded in generalization bounds.
Empirical results on seven public NLP benchmarks demonstrate that ESPO outperforms the state-of-the-art GEPA by an average of 3.76 percentage points in accuracy, while producing prompts nearly half as long. Notably, on GSM8K, accuracy jumps from 15% to 91.4%. Cross-model experiments across four different student models confirm the robustness and transferability of the method. Ablation studies reveal that diversity alone can hurt performance, emphasizing the importance of structured diagnosis and stable selection.
Overall, ESPO offers a principled, efficient, and effective approach to prompt optimization. Its theoretical foundations and empirical success suggest broad applicability in NLP tasks, paving the way for more reliable and scalable prompt engineering. Future work will aim to reduce computational costs, improve error diagnosis, and extend the framework to multi-modal and industrial applications, further advancing the automation of prompt design in AI systems.
Deep Analysis
Background
近年来，随着大规模预训练模型（如GPT、BERT）的广泛应用，提示工程逐渐成为提升模型性能的重要手段。早期方法多依赖手工设计，效率低且难以适应多变任务。自动提示优化算法如APE、OPRO、GEPA等相继出现，试图通过搜索策略自动生成高质量提示。GEPA采用演化算法，通过不断变异和选择优化提示，但存在提示膨胀、搜索空间有限、候选不稳定等问题。近年来，贝叶斯、反思和多目标优化等技术被引入，试图解决这些难题，但仍未根本突破提示过度膨胀和泛化不足的瓶颈。本文提出的ESPO旨在通过结构化误差诊断和统计估计，提供一种系统化的解决方案，推动提示优化的理论与实践发展。
Core Problem
现有演化算法在提示优化中面临多重挑战：一是提示膨胀严重，导致提示变得冗长且难以理解；二是搜索多样性不足，容易陷入局部最优；三是候选提示选择不稳定，受验证集噪声影响大。这些问题限制了提示的效果和模型的泛化能力。尤其是在小验证集条件下，随机噪声可能导致误导性选择，影响最终性能。如何系统性识别错误根源、生成多样化且简洁的候选提示，并稳健地选择最优提示，成为亟待解决的核心难题。
Innovation
ESPO的创新点包括：1）引入结构化误差诊断，利用大模型识别训练中的错误簇，确保一次性覆盖所有根本性错误；2）结合多策略（修正、合并、消融、事实注入）生成多样化提示，避免偏见和膨胀，提升搜索效率；3）采用bootstrap重采样进行稳定性筛选，确保所选提示在不同验证样本中表现稳健，理论上由泛化界保证。该框架将演化搜索转变为统计估计，显著提升优化的效率和效果，超越传统方法。
Methodology
- �� 误差诊断：利用大模型反思能力，将训练错误簇集成，识别根本性错误模式；• 候选生成：基于诊断结果，通过四种策略（修正、合并、消融、事实注入）生成多样提示，避免偏见和膨胀；• 组合候选：多轮“授粉”操作，从不同候选中生成新提示，丰富搜索空间；• 自助选择：多次bootstrap重采样验证集，筛选在多次采样中表现最稳健的提示，确保泛化。整个流程由理论支撑，逐步缩小训练-测试差距。
Experiments
在七个公开数据集（如Tweet、MMLU、GSM8K）上，采用从弱提示开始，比较ESPO与GEPA、COPRO、MIPROv2等基线。指标包括准确率、提示长度和推理速度。参数K（候选策略数）、B（bootstrap次数）和m（诊断批次大小）调优。还进行了跨模型验证，使用不同规模的学生模型，验证泛化能力。消融实验验证各策略贡献，分析参数敏感性。
Results
ESPO在所有数据集上均优于GEPA，平均提升3.76个百分点，达到74.67%。提示长度缩短47%，推理速度提升显著。Qwen3 GSM8K从15%提升至91.4%，验证误差簇诊断的有效性。消融实验显示：引入多样性（K策略）未加bootstrap反而降低性能，结构化误差诊断和稳定选择是关键。跨模型验证表明，ESPO的优势具有良好的迁移性。
Applications
该方法适用于需要高效提示优化的场景，如自动问答、对话系统、任务指令生成等。只需少量验证样本，即可显著提升模型性能和效率。未来，结合多模型反思和强化学习，将推动提示优化的自动化，广泛应用于工业界，提升大模型的智能化水平。
Limitations & Outlook
ESPO的计算成本较高，候选生成和自助采样阶段需多次模型推理，限制大规模应用。误差簇的诊断依赖反思模型能力；复杂或微妙的错误簇可能影响诊断效果。假设误差簇之间相互独立，实际中存在相关性，可能影响理论保证。未来需优化效率、提升诊断精度，并扩展多模型融合能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，目标是做出一道完美的菜。传统方法就像不断试错，逐步调整调料，结果可能越调越复杂，甚至变得难以理解。ESPO像是请厨师先观察菜的味道，找出哪些调料导致问题，然后用不同的调料组合试验，最后用多次尝试确认哪种调料最稳妥。这样一来，不仅菜变得更好吃，还能用更少的调料做出更好的菜。它用一种聪明的方式，系统性地找到最合适的调料组合，避免无用的复杂和浪费。
ELI14 Explained like you're 14
想象你在学校里准备一份演讲，但刚开始的稿子写得很糟糕。传统的方法就像反复自己改，可能会越改越乱。ESPO就像请老师帮你先找出稿子中哪些部分出错了，然后用不同的方法改稿，比如删掉多余的内容、加入新信息、简化句子。最后，你用多次模拟演讲，看看哪份稿子听起来最自然、最有说服力。这样，你就能用更少的内容，讲得更清楚、更有趣。它让你用科学的方法，找到最棒的演讲稿，省时又有效。
Glossary
Prompt Optimization（提示优化）
通过算法自动改进模型输入指令，以提升输出质量。涉及搜索和优化机制。
本文旨在通过结构化方法提升提示效果。
Error Clustering（误差簇集）
用大模型识别训练中错误的根源，将错误分组以指导优化。
诊断阶段的关键步骤。
Bootstrap Stability Selection（自助稳定选择）
多次重采样验证，筛选出在不同样本中表现稳定的提示。
确保提示在不同数据扰动下的稳健性。
Multi-Strategy Candidate Generation（多策略候选生成）
结合多种方法生成多样化提示，避免偏见。
提升搜索空间丰富性。
Generalization Bound（泛化界）
理论上限制模型在新数据上的表现差距。
为ESPO的效果提供理论保障。
Open Questions Unanswered questions from this research
- 1 如何降低ESPO的计算成本以适应大规模应用？
- 2 误差簇诊断在复杂错误场景中的准确性如何提升？
- 3 多模型融合能否增强误差簇识别的鲁棒性？
Applications
Immediate Applications
自动问答系统
用ESPO优化提示，提高问答准确率，减少提示长度，提升响应速度。
多任务指令生成
自动生成高质量指令，提升多任务模型表现，降低人工调试成本。
Long-term Vision
智能提示平台
构建全自动提示优化系统，支持多模型、多场景应用，推动工业界AI普及。
Abstract
Evolutionary prompt optimizers such as GEPA suffer from prompt bloat: each iteration appends rules and caveats, producing prompts up to 3$\times$ longer yet no more accurate. We trace this to three deficiencies - incomplete error observation, limited search diversity, and unreliable selection - and propose ESPO (Error-Structured Prompt Optimization), which decomposes prompt optimization into three phases: Diagnose clusters all training errors into structural patterns in one round; Propose generates candidates via four complementary strategies with independent biases; Select applies bootstrap stability selection. On seven public NLP benchmarks - Tweet, MMLU, GSM8K, HotpotQA, ScoNe, HoVer, and PUPA - ESPO improves average accuracy by $+$3.76 pp over the state-of-the-art (74.67% vs 70.91% for GEPA), matching or exceeding GEPA on every dataset while producing prompts 47% shorter (1,004 vs 1,878 chars) and faster at inference. Cross-model experiments across four additional student models (Gemma 3 12B, Mistral 14B, Qwen3 32B, Claude Haiku 4.5) show ESPO yields the best average accuracy on every model tested, with the largest gap on Qwen3 GSM8K (15.00% $\to$ 91.40%). A generalization bound (Appendix) grounds each phase in a corresponding term of the test-time gap, and the ablation confirms a key prediction: adding diversity without bootstrap selection actually hurts performance ($-$1.20%).