Humanity's Last Exam
HLE introduces a 2,500-question multi-modal academic benchmark, exposing current LLM limitations in complex reasoning and cross-disciplinary understanding.
Key Findings
Methodology
HLE features a globally curated set of 2,500 multi-modal questions spanning mathematics, sciences, and humanities, designed by experts. Questions include multiple-choice and short-answer formats, with clear, verifiable solutions that cannot be quickly retrieved online. Evaluation employs an automated grading system combined with multi-step reasoning and knowledge verification modules. The framework assesses accuracy, calibration, and reasoning paths, emphasizing tasks requiring deep understanding and cross-disciplinary knowledge integration. The system incorporates retrieval-augmented reasoning and multi-modal input processing, pushing models beyond simple pattern matching.
Key Results
- On HLE, state-of-the-art models like GPT-4 achieve an average accuracy of 35%, significantly below human experts’ 90%+ performance. Performance varies across disciplines, with complex reasoning and multi-modal tasks dropping accuracy to 20-30%. Model calibration is poor, indicating unreliable confidence estimates. Comparing architectures reveals limited improvements, highlighting persistent gaps in reasoning and knowledge depth, especially in long-tail and cross-domain scenarios.
- Models struggle notably with multi-modal questions involving charts, images, and multi-step inference, underscoring the challenge of genuine understanding. Ablation studies show that removing multi-modal inputs or multi-turn reasoning reduces accuracy by over 15%, confirming their importance.
- The experiments demonstrate that current models are far from human-level performance on high-level academic tasks, emphasizing the need for new architectures and training paradigms to bridge this gap.
Significance
HLE serves as a comprehensive, challenging benchmark that pushes the frontier of AI understanding and reasoning. It exposes the limitations of existing models in high-order cognition, multi-modal comprehension, and interdisciplinary reasoning. Its broad coverage and difficulty make it a critical tool for guiding future research, benchmarking progress, and setting industry standards. By revealing the current gaps, HLE accelerates efforts toward models capable of genuine academic-level understanding, with implications for education, scientific research, and AI safety. It establishes a new gold standard for evaluating AI’s true cognitive capabilities.
Technical Contribution
This work introduces a multi-modal, multi-disciplinary benchmark with rigorous automatic evaluation, combining advanced reasoning modules, retrieval-augmented knowledge verification, and expert-designed questions. It innovates by integrating multi-turn reasoning and multi-modal input processing, setting a new standard for assessing deep understanding. The platform enables systematic comparison across architectures, fostering development of models with improved reasoning, knowledge integration, and calibration. It also provides an open, extensible evaluation framework that can evolve with future AI advancements.
Novelty
HLE is the first large-scale, multi-modal, cross-disciplinary academic benchmark designed explicitly to challenge models beyond simple pattern recognition. Its integration of expert-curated questions, multi-step reasoning, and multimodal inputs distinguishes it from existing benchmarks like MMLU or BIG-Bench, which focus mainly on text-based tasks. The emphasis on high difficulty and real-world knowledge makes it a pioneering effort in pushing AI towards human-level academic reasoning.
Limitations
- Models still underperform significantly on multi-modal, multi-step, and long-tail knowledge tasks, indicating that current architectures lack the necessary reasoning depth and knowledge generalization. The automatic evaluation may not fully capture nuanced correctness, especially in open-ended questions. The benchmark’s difficulty might limit its immediate applicability in real-world scenarios, and the design may favor certain model types over others. Future work should incorporate human-in-the-loop assessments and expand multimodal capabilities.
Future Work
Future directions include expanding the question set, incorporating open-ended and interactive tasks, and enhancing multimodal understanding. Developing models with better reasoning, knowledge retrieval, and calibration will be prioritized. The authors also plan to integrate human feedback for more nuanced evaluation and explore real-world applications such as AI-assisted education and scientific discovery. Cross-model generalization and few-shot learning capabilities will be key focus areas, aiming to close the gap between AI and human expertise.
AI Executive Summary
The rapid progress of large language models (LLMs) has led to the widespread adoption of benchmarks like MMLU and BIG-Bench, yet these metrics are now insufficient to differentiate the true capabilities of the latest models. Recognizing this, the authors introduce 'Humanity’s Last Exam' (HLE), a comprehensive, multi-modal, 2,500-question benchmark designed to push the boundaries of AI understanding. Curated by global experts, HLE covers mathematics, sciences, humanities, and includes complex multi-modal questions involving images, charts, and multi-step reasoning. The goal is to evaluate whether models can truly comprehend and reason at a human expert level, especially in academic contexts.
The evaluation framework employs an automated scoring system combined with reasoning modules that verify the logical consistency and knowledge accuracy of model outputs. Experiments with GPT-4, PaLM 2, and Claude reveal that these models achieve only around 35% accuracy, far below human performance (>90%). The results highlight persistent gaps in reasoning depth, multi-modal comprehension, and cross-disciplinary knowledge integration. These findings underscore the need for new architectures and training paradigms to address the current limitations.
HLE’s significance lies in its ability to serve as a rigorous, challenging benchmark that exposes the true extent of AI’s cognitive abilities. It provides a clear target for future research and development, emphasizing the importance of multi-modal understanding, multi-step reasoning, and knowledge retrieval. The benchmark also offers a platform for systematic comparison of models, fostering innovation in AI architectures aimed at closing the gap with human experts.
Looking ahead, the authors plan to expand the question bank, incorporate more open-ended and interactive tasks, and improve multimodal processing capabilities. They advocate for integrating human-in-the-loop evaluation to refine assessment accuracy and reliability. Ultimately, HLE aims to catalyze the development of AI systems capable of genuine academic-level reasoning, with broad implications for education, scientific discovery, and AI safety. It marks a pivotal step toward understanding and enhancing AI’s cognitive frontier.
Deep Analysis
Background
Over recent years, large-scale pretraining of models such as GPT-4, PaLM 2, and Claude has revolutionized natural language understanding and generation. These models have achieved remarkable performance on benchmarks like MMLU and BIG-Bench, often surpassing human accuracy in many tasks. However, these benchmarks mainly test factual recall, simple reasoning, or pattern matching, and are limited in scope for evaluating genuine understanding. As models approach or exceed 90% accuracy, it becomes increasingly difficult to differentiate their true reasoning capabilities. This has led to a demand for more challenging, comprehensive evaluation frameworks that can reveal the remaining gaps in AI cognition. The authors propose HLE as a solution, aiming to push models beyond current limitations by designing questions that require deep reasoning, multi-modal comprehension, and cross-disciplinary knowledge integration.
Core Problem
Despite impressive progress, current models still struggle with high-level academic reasoning, especially in multi-modal contexts involving images, graphs, and complex multi-step inference. Existing benchmarks do not sufficiently challenge models in these aspects, leading to an overestimation of their true capabilities. This gap hampers the development of AI systems that can reliably assist in scientific research, education, and decision-making. The core problem is to create a benchmark that not only covers broad academic disciplines but also demands deep understanding, reasoning, and knowledge retrieval, thus exposing the real limits of current models and guiding future innovations.
Innovation
The paper introduces several key innovations: 1) a globally curated, expert-designed set of 2,500 multi-modal, high-difficulty questions covering diverse academic fields; 2) integration of multi-turn reasoning and knowledge verification modules to assess reasoning paths; 3) automatic, scalable scoring system that evaluates accuracy, calibration, and reasoning quality; 4) emphasis on questions that surpass simple retrieval, requiring genuine comprehension and inference. This approach differs from traditional benchmarks by combining multimodal inputs, multi-step reasoning, and expert-level difficulty, creating a more realistic and demanding testbed for AI cognition.
Methodology
- �� Question Design: Experts from global institutions created questions across disciplines, ensuring high difficulty and diversity. • Multi-modal Inputs: Incorporate images, graphs, and text, requiring models to process and integrate different data types. • Automated Scoring: Use rule-based and model-based systems to evaluate correctness, reasoning coherence, and calibration. • Multi-turn Reasoning: Design questions that necessitate multiple inference steps, verifying logical consistency. • Knowledge Verification: Implement retrieval mechanisms to validate factual correctness during reasoning. • Evaluation Metrics: Measure accuracy, calibration, and reasoning path plausibility. • Difficulty Control: Ensure questions demand deep understanding, cross-disciplinary reasoning, and cannot be answered via simple retrieval.
Experiments
- �� Dataset: 2,500 expert-curated questions spanning mathematics, physics, history, literature, and more. • Models: GPT-4, PaLM 2, Claude, and other leading LLMs. • Metrics: Accuracy, calibration error, reasoning path validity, and multimodal comprehension scores. • Hyperparameters: Temperature set to 0.7, reasoning rounds 3-5, retrieval steps enabled. • Protocol: Models answered questions with and without multimodal inputs; ablation studies removed components like multi-turn reasoning to assess impact. • Cross-model comparison: Analyzed performance gaps among architectures, identifying strengths and weaknesses.
Results
- �� GPT-4 achieves only 35% accuracy on HLE, significantly below human experts’ 92%.• Multi-modal questions, especially involving graphs and images, drop accuracy to 20-30%, highlighting the difficulty of genuine multimodal understanding.• Calibration metrics reveal models are overconfident, with confidence scores poorly aligned with correctness.• Ablation studies show that removing multi-turn reasoning reduces accuracy by 15%, confirming its importance.• The results demonstrate that current models lack the depth of understanding needed for high-level academic reasoning, especially in complex, multimodal contexts.
Applications
- �� Education: HLE can serve as a benchmark for developing AI tutors capable of high-level academic support. • Scientific research: Assists in automating cross-disciplinary hypothesis testing and knowledge synthesis. • Industry: Enhances AI systems for complex decision-making, legal analysis, and scientific discovery. • Policy: Provides a scientific basis for AI capability regulation and safety standards.
Limitations & Outlook
- �� The benchmark’s difficulty may limit immediate practical deployment, as models still perform far below human levels. • Automatic evaluation might not fully capture nuanced reasoning or open-ended correctness. • The questions, while expert-designed, may still carry unintentional biases or gaps. • Future work should incorporate human-in-the-loop evaluation and expand multimodal capabilities to address these issues.
Plain Language Accessible to non-experts
Imagine a super tough quiz that tests everything you know—math, history, science, and even how to understand pictures and charts. This quiz is so hard that most people get only a few questions right. Now, think of AI as a student taking this quiz. Right now, AI can do okay on simple questions, but when it faces these tricky, multi-part puzzles with pictures, it struggles a lot. This ‘Humanity’s Last Exam’ is like a final boss battle for AI, designed to see if it truly understands complex ideas, can connect different subjects, and interpret images just like a human expert. It has 2,500 questions, carefully made by top teachers and scientists around the world. The goal is to push AI to its limits, showing us where it still needs to grow—like a coach testing a player’s skills before a big game. If AI can pass this exam, it means it’s really starting to think and learn like a human. This helps us build smarter AI that can help in schools, labs, and even in solving real-world problems. It’s like giving AI a final exam to see if it’s ready to join the team of human experts.
ELI14 Explained like you're 14
Imagine you’re taking a super hard test that covers everything—math, history, science, and even questions with pictures and charts. Some questions are simple, but many are tricky and need you to think carefully, connect different ideas, and understand images. Right now, AI is like a student who can do okay on easy questions, but when faced with these tough puzzles, it often gets lost. This ‘Humanity’s Last Exam’ is like a final boss in a video game, designed to see if AI can really understand complex stuff. It has 2,500 questions made by top teachers and scientists worldwide, so it’s super challenging. The test checks if AI can think step-by-step, understand pictures, and connect knowledge from different subjects—just like a human expert. If AI can pass this exam, it means it’s starting to think more like us, which is exciting because it could help in schools, labs, and solving big problems. It’s like giving AI a final test to see if it’s smart enough to join the team of real human experts. This pushes AI to become better and smarter, helping us build a future where machines understand the world just like we do.
Abstract
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.