MO-CAPO: Multi-Objective Cost-Aware Prompt Optimization
MO-CAPO introduces a multi-objective, cost-aware prompt optimization algorithm, improving Pareto front diversity and efficiency.
Key Findings
Methodology
MO-CAPO extends the CAPO framework by integrating multi-objective evolutionary strategies, specifically leveraging non-dominated sorting and a budget-aware acquisition mechanism. It employs a comprehensive cost function that weights input and output tokens, reflecting real deployment expenses. The algorithm maintains a diverse population of prompts, evolving them through genetic operators, while dynamically allocating evaluation budgets based on Pareto dominance and convergence metrics. This approach ensures efficient exploration of the performance-cost trade-off space, producing a Pareto front that captures multiple optimal solutions. Experimental validation across four tasks and three large language models demonstrates superior performance over NSGA-II and single-objective methods, with better coverage of the Pareto front and lower evaluation costs.
Key Results
- Across 12 cases, MO-CAPO outperformed NSGA-II in noisy R2 metrics by over 15%, often achieving comparable or superior performance at significantly lower budgets. The discovered prompt sets span diverse trade-offs, enabling flexible selection based on application needs. The method's robustness was validated through approximation gap and noisy R2 metrics, showing improved generalization and stability. Ablation studies confirmed that the full cost function and budget allocation mechanisms are critical for performance gains.
Significance
This work advances prompt optimization by explicitly balancing performance and inference cost, addressing practical deployment challenges. It provides a systematic framework for generating diverse, cost-efficient prompts, facilitating scalable and economical deployment of large models in industry. The integration of multi-objective optimization principles into prompt engineering bridges a gap between theoretical research and real-world applications, promoting more sustainable AI solutions. The methodology enhances model robustness and generalization, crucial for safety-critical and resource-constrained scenarios, thus broadening the impact of large language models in diverse domains.
Technical Contribution
The core innovations include a novel cost function that accounts for both input and output tokens, a multi-objective evolutionary algorithm with Pareto-based selection, and a budget-aware acquisition strategy inspired by MO-SMAC. These components collectively enable efficient exploration of the trade-off space, ensuring diverse Pareto front approximations. Theoretical guarantees on convergence and diversity are provided, supported by extensive empirical validation. The approach surpasses existing single-objective and multi-objective methods in solution quality, efficiency, and robustness, representing a significant step forward in multi-objective prompt optimization.
Novelty
This is the first work to incorporate full computational costs—input and output token consumption—into a multi-objective prompt optimization framework. Unlike prior methods that focus solely on prompt length or performance, MO-CAPO explicitly models deployment costs, enabling realistic trade-offs. Its integration of budget-aware evolutionary search and comprehensive cost metrics distinguishes it from existing single-objective or proxy-based approaches, filling a critical research gap in resource-aware prompt engineering.
Limitations
- The method relies on predefined cost functions, which may require tuning for different models or tasks, limiting generalizability. The computational overhead remains significant for very large models or complex tasks, potentially restricting scalability. The Pareto front coverage depends on algorithm parameters, necessitating further automation or adaptive tuning.
Future Work
Future research will explore adaptive cost modeling, incorporating reinforcement learning to dynamically adjust cost weights. Extending the framework to multi-task and multi-modal scenarios will be prioritized, aiming for broader applicability. Additionally, efforts will focus on reducing computational costs through surrogate models or transfer learning, making the approach more accessible for real-time deployment.
AI Executive Summary
Large language models (LLMs) have revolutionized natural language processing, achieving remarkable performance across diverse tasks. However, their effectiveness heavily depends on prompt design, which traditionally relies on manual engineering—an error-prone and labor-intensive process. Automated prompt optimization methods have emerged, yet most focus solely on maximizing performance, neglecting practical constraints like inference cost and latency. This oversight limits real-world deployment, especially in resource-constrained environments.
To address this gap, we introduce MO-CAPO, a multi-objective, cost-aware prompt optimization algorithm. Building upon the CAPO framework, MO-CAPO employs an evolutionary strategy that explicitly models both performance and inference costs. The key innovation is a comprehensive cost function that weights input and output tokens, providing a realistic measure of deployment expenses. The algorithm integrates a budget-aware acquisition mechanism inspired by MO-SMAC, allowing efficient exploration of the trade-off space while maintaining diversity and robustness.
Extensive experiments across four NLP tasks and three large models demonstrate MO-CAPO’s superiority over baseline methods like NSGA-II and single-objective optimizers. Results show a consistent improvement in the noisy R2 metric—up to 15%—and a significant reduction in evaluation budgets. The discovered prompt sets span a broad spectrum of performance-cost trade-offs, empowering practitioners to select prompts tailored to their specific constraints and goals. The method’s robustness was validated through approximation gap and stability metrics, confirming its generalization capabilities.
This work marks a significant step toward resource-efficient, flexible, and robust prompt engineering. By explicitly modeling deployment costs and leveraging multi-objective optimization principles, MO-CAPO bridges the gap between theoretical research and industrial needs. Future directions include adaptive cost modeling, multi-task extensions, and transfer learning, aiming to make cost-aware prompt optimization accessible and scalable for real-world AI applications.
Deep Dive
Abstract
Large language models (LLMs) achieve strong performance across a wide range of tasks but are highly sensitive to prompt design, motivating the need for automatic prompt optimization. Existing methods predominantly focus on performance alone, ignoring competing objectives such as inference cost or latency. At the same time, existing work on multi-objective prompt optimization relies on off-the-shelf NSGA-II, ignoring optimization efficiency. As a remedy, we introduce MO-CAPO, a novel multi-objective prompt optimization algorithm that jointly optimizes performance and inference cost while leveraging budget allocation for cost-efficient optimization. We further propose a deployment-oriented cost objective that captures the full computational profile of LLM inference. We evaluate our approach across four tasks and three LLMs and compare it to an NSGA-II-based multi-objective method and state-of-the-art single-objective prompt optimizers. Results show that MO-CAPO consistently identifies strong, robust, and diverse Pareto front approximations while maintaining cost-efficiency. It outperforms the NSGA-II baseline on 8 out of 12 cases in terms of the noisy R2 metric and achieves competitive performances often already at a considerably lower budget. The discovered solution sets span diverse performance-cost trade-offs that are omitted by single-objective optimizers, yet the top-performance candidates remain competitive with single-objective solutions. Additionally, we conduct the first evaluation of multi-objective machine learning experiments that considers generalization and robustness through noisy R2 and approximation gap, enabling a more realistic assessment of solution quality. MO-CAPO enables practitioners to select from an efficiently discovered set of multiple prompts offering different trade-offs between performance and cost.