EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks

The evolution of large language models (LLMs) such as GPT-4, Claude, and Mistral has revolutionized NLP tasks, achieving impressive results on benchmarks like SuperGLUE and MMLU. However, these benchmarks predominantly rely on objective, well-defined tasks with clear correct answers, such as multiple-choice questions or translation. In real-world professional environments—finance, law, management—tasks are often open-ended, requiring nuanced judgment, contextual understanding, and domain expertise. Existing evaluation methods, mainly automated metrics, fail to capture these complexities, leading to a gap between benchmark performance and practical utility. Prior work has attempted automatic or crowd-sourced evaluation, but these approaches lack the precision and reliability needed for high-stakes decision-making. Recognizing this, the present study constructs EuroExec, a benchmark grounded in real industry scenarios, evaluated by domain experts, to better reflect the true capabilities of frontier models in professional contexts.

Despite rapid advancements, current LLMs demonstrate limited competence in complex decision-making tasks that require professional judgment, especially in European corporate settings. Automated metrics such as BLEU and ROUGE are insufficient for subjective assessments, and existing benchmarks do not adequately reflect real-world challenges. The core problem is how to reliably evaluate whether models can meet the standards of human experts in nuanced, open-ended tasks involving strategic, legal, or financial considerations. This gap hampers the deployment of AI in critical decision-making roles, risking overestimation of model capabilities and underestimating their limitations. Addressing this requires a rigorous, human-centered evaluation framework that captures the subtleties of expert judgment.

This work introduces several innovations: 1) EuroExec, a benchmark with 413 real-world European executive tasks authored by industry experts; 2) a comprehensive evaluation protocol combining multi-attribute rubrics, item-specific checklists, and preference rankings; 3) rigorous statistical analysis confirming evaluator consistency; 4) systematic comparison of six frontier models, revealing significant performance gaps. Unlike prior benchmarks relying solely on automatic metrics, this approach emphasizes human expert judgment, providing a more accurate reflection of model readiness for professional use. The integration of real industry scenarios and expert evaluations marks a significant step forward in AI benchmarking for complex, subjective tasks.

• �� Task design: Experts from finance, marketing, business, and product domains crafted 413 long-form decision questions based on real cases, ensuring relevance and complexity.
• �� Validation: Questions underwent automated QA checks for relevance and safety, followed by difficulty gating using two different LLMs to filter out overly simple or irrelevant tasks.
• �� Response generation: Six frontier models (e.g., Fable 5, GPT-5.5) generated answers via API, with responses kept context-free to ensure fairness.
• �� Evaluation: Experts independently scored responses using a rubric (attributes: Domain, Localization, Reasoning, Communication, Actionability), checklists, and preference rankings. Responses were anonymized and shuffled.
• �� Statistical analysis: Employed Kendall τ and p-values to verify evaluator consistency and significance of differences. The 'Solve Rate' metric was derived from rubric scores and checklist fulfillment, with a threshold of ≥3.0 and ≥60% respectively to consider a task solved.

The experimental setup involved diverse real-world tasks across four domains, with models evaluated on their ability to produce relevant, accurate, and actionable responses. The expert evaluation process was intensive, with two experts independently scoring each response, ensuring high reliability. The analysis revealed that even the best models achieved a maximum Solve Rate of 56.9%, indicating substantial room for improvement. The performance was stratified by task difficulty, showing a consistent decline as complexity increased. The study also compared automatic metrics with human judgments, finding weak correlations, thus emphasizing the importance of human evaluation in assessing real-world applicability. Additional ablation studies confirmed the robustness of the evaluation protocol.

The primary outcome was that no model exceeded a 57% solve rate, with GPT-5.5 at 51.3%. Expert answers scored near perfect, with 74% preference over model responses. Performance degraded significantly on harder tasks, with the strongest models barely surpassing 50% in easy questions. Statistical tests confirmed evaluator consistency and the significance of model rankings. Automatic metrics showed poor correlation with human scores, underscoring their limitations. The results highlight that current models, despite progress, are far from meeting professional standards in complex decision scenarios.

This benchmark can guide industry practitioners in selecting and fine-tuning AI tools for high-stakes decision-making, such as legal analysis, financial planning, and strategic management. It also informs AI developers about critical weaknesses to address, especially reasoning and domain-specific knowledge. Long-term, the framework encourages integrating expert judgment into model training and evaluation, fostering AI systems capable of supporting real-world professional tasks with reliability and accountability.

The evaluation relies heavily on subjective expert judgment, which, despite statistical validation, may introduce biases. The sample size, though large, is limited to specific domains and task types, restricting broader applicability. Responses are assessed without contextual information, which may differ from real-world scenarios. Computational costs of expert evaluation are high, posing scalability challenges. Future work should focus on automating parts of the evaluation while maintaining accuracy, expanding task diversity, and incorporating contextual data to better simulate real-world conditions.