Humanity's Last Exam

TL;DR

提出Humanity's Last Exam(HLE),涵盖2500题多模态学术题,挑战当前大模型能力。

cs.LG 🔴 高级 2025-01-24 55 次浏览
Long Phan Alice Gatti Ziwen Han Nathaniel Li Josephina Hu Hugh Zhang Chen Bo Calvin Zhang Mohamed Shaaban John Ling Sean Shi Michael Choi Anish Agrawal Arnav Chopra Adam Khoja Ryan Kim Richard Ren Jason Hausenloy Oliver Zhang Mantas Mazeika Dmitry Dodonov Tung Nguyen Jaeho Lee Daron Anderson Mikhail Doroshenko Alun Cennyth Stokes Mobeen Mahmood Oleksandr Pokutnyi Oleg Iskra Jessica P. Wang John-Clark Levin Mstyslav Kazakov Fiona Feng Steven Y. Feng Haoran Zhao Michael Yu Varun Gangal Chelsea Zou Zihan Wang Serguei Popov Robert Gerbicz Geoff Galgon Johannes Schmitt Will Yeadon Yongki Lee Scott Sauers Alvaro Sanchez Fabian Giska Marc Roth Søren Riis Saiteja Utpala Noah Burns Gashaw M. Goshu Mohinder Maheshbhai Naiya Chidozie Agu Zachary Giboney Antrell Cheatom Francesco Fournier-Facio Sarah-Jane Crowson Lennart Finke Zerui Cheng Jennifer Zampese Ryan G. Hoerr Mark Nandor Hyunwoo Park Tim Gehrunger Jiaqi Cai Ben McCarty Alexis C Garretson Edwin Taylor Damien Sileo Qiuyu Ren Usman Qazi Lianghui Li Jungbae Nam John B. Wydallis Pavel Arkhipov Jack Wei Lun Shi Aras Bacho Chris G. Willcocks Hangrui Cao Sumeet Motwani Emily de Oliveira Santos Johannes Veith Edward Vendrow Doru Cojoc Kengo Zenitani Joshua Robinson Longke Tang Yuqi Li Joshua Vendrow Natanael Wildner Fraga Vladyslav Kuchkin Andrey Pupasov Maksimov Pierre Marion Denis Efremov Jayson Lynch Kaiqu Liang Aleksandar Mikov Andrew Gritsevskiy Julien Guillod Gözdenur Demir Dakotah Martinez Ben Pageler Kevin Zhou Saeed Soori Ori Press Henry Tang Paolo Rissone Sean R. Green Lina Brüssel Moon Twayana Aymeric Dieuleveut Joseph Marvin Imperial Ameya Prabhu Jinzhou Yang Nick Crispino Arun Rao Dimitri Zvonkine Gabriel Loiseau Mikhail Kalinin Marco Lukas Ciprian Manolescu Nate Stambaugh Subrata Mishra Tad Hogg Carlo Bosio Brian P Coppola Julian Salazar Jaehyeok Jin Rafael Sayous Stefan Ivanov Philippe Schwaller Shaipranesh Senthilkuma Andres M Bran Andres Algaba Kelsey Van den Houte Lynn Van Der Sypt Brecht Verbeken David Noever Alexei Kopylov Benjamin Myklebust Bikun Li Lisa Schut Evgenii Zheltonozhskii Qiaochu Yuan Derek Lim Richard Stanley Tong Yang John Maar Julian Wykowski Martí Oller Anmol Sahu Cesare Giulio Ardito Yuzheng Hu Ariel Ghislain Kemogne Kamdoum Alvin Jin Tobias Garcia Vilchis Yuexuan Zu Martin Lackner James Koppel Gongbo Sun Daniil S. Antonenko Steffi Chern Bingchen Zhao Pierrot Arsene Joseph M Cavanagh Daofeng Li Jiawei Shen Donato Crisostomi Wenjin Zhang Ali Dehghan Sergey Ivanov David Perrella Nurdin Kaparov Allen Zang Ilia Sucholutsky Arina Kharlamova Daniil Orel Vladislav Poritski Shalev Ben-David Zachary Berger Parker Whitfill Michael Foster Daniel Munro Linh Ho Shankar Sivarajan Dan Bar Hava Aleksey Kuchkin David Holmes Alexandra Rodriguez-Romero Frank Sommerhage Anji Zhang Richard Moat Keith Schneider Zakayo Kazibwe Don Clarke Dae Hyun Kim Felipe Meneguitti Dias Sara Fish Veit Elser Tobias Kreiman Victor Efren Guadarrama Vilchis Immo Klose Ujjwala Anantheswaran Adam Zweiger Kaivalya Rawal Jeffery Li Jeremy Nguyen Nicolas Daans Haline Heidinger Maksim Radionov Václav Rozhoň Vincent Ginis Christian Stump Niv Cohen Rafał Poświata Josef Tkadlec Alan Goldfarb Chenguang Wang Piotr Padlewski Stanislaw Barzowski Kyle Montgomery Ryan Stendall Jamie Tucker-Foltz Jack Stade T. Ryan Rogers Tom Goertzen Declan Grabb Abhishek Shukla Alan Givré John Arnold Ambay Archan Sen Muhammad Fayez Aziz Mark H Inlow Hao He Ling Zhang Younesse Kaddar Ivar Ängquist Yanxu Chen Harrison K Wang Kalyan Ramakrishnan Elliott Thornley Antonio Terpin Hailey Schoelkopf Eric Zheng Avishy Carmi Ethan D. L. Brown Kelin Zhu Max Bartolo Richard Wheeler Martin Stehberger Peter Bradshaw JP Heimonen Kaustubh Sridhar Ido Akov Jennifer Sandlin Yury Makarychev Joanna Tam Hieu Hoang David M. Cunningham Vladimir Goryachev Demosthenes Patramanis Michael Krause Andrew Redenti David Aldous Jesyin Lai Shannon Coleman Jiangnan Xu Sangwon Lee Ilias Magoulas Sandy Zhao Ning Tang Michael K. Cohen Orr Paradise Jan Hendrik Kirchner Maksym Ovchynnikov Jason O. Matos Adithya Shenoy Michael Wang Yuzhou Nie Anna Sztyber-Betley Paolo Faraboschi Robin Riblet Jonathan Crozier Shiv Halasyamani Shreyas Verma Prashant Joshi Eli Meril Ziqiao Ma Jérémy Andréoletti Raghav Singhal Jacob Platnick Volodymyr Nevirkovets Luke Basler Alexander Ivanov Seri Khoury Nils Gustafsson Marco Piccardo Hamid Mostaghimi Qijia Chen Virendra Singh Tran Quoc Khánh Paul Rosu Hannah Szlyk Zachary Brown Himanshu Narayan Aline Menezes Jonathan Roberts William Alley Kunyang Sun Arkil Patel Max Lamparth Anka Reuel Linwei Xin Hanmeng Xu Jacob Loader Freddie Martin Zixuan Wang Andrea Achilleos Thomas Preu Tomek Korbak Ida Bosio Fereshteh Kazemi Ziye Chen Biró Bálint Eve J. Y. Lo Jiaqi Wang Maria Inês S. Nunes Jeremiah Milbauer M Saiful Bari Zihao Wang Behzad Ansarinejad Yewen Sun Stephane Durand Hossam Elgnainy Guillaume Douville Daniel Tordera George Balabanian Hew Wolff Lynna Kvistad Hsiaoyun Milliron Ahmad Sakor Murat Eron Andrew Favre D. O. Shailesh Shah Xiaoxiang Zhou Firuz Kamalov Sherwin Abdoli Tim Santens Shaul Barkan Allison Tee Robin Zhang Alessandro Tomasiello G. Bruno De Luca Shi-Zhuo Looi Vinh-Kha Le Noam Kolt Jiayi Pan Emma Rodman Jacob Drori Carl J Fossum Niklas Muennighoff Milind Jagota Ronak Pradeep Honglu Fan Jonathan Eicher Michael Chen Kushal Thaman William Merrill Moritz Firsching Carter Harris Stefan Ciobâcă Jason Gross Rohan Pandey Ilya Gusev Adam Jones Shashank Agnihotri Pavel Zhelnov Mohammadreza Mofayezi Alexander Piperski David K. Zhang Kostiantyn Dobarskyi Roman Leventov Ignat Soroko Joshua Duersch Vage Taamazyan Andrew Ho Wenjie Ma William Held Ruicheng Xian Armel Randy Zebaze Mohanad Mohamed Julian Noah Leser Michelle X Yuan Laila Yacar Johannes Lengler Katarzyna Olszewska Claudio Di Fratta Edson Oliveira Joseph W. Jackson Andy Zou Muthu Chidambaram Timothy Manik Hector Haffenden Dashiell Stander Ali Dasouqi Alexander Shen Bita Golshani David Stap Egor Kretov Mikalai Uzhou Alina Borisovna Zhidkovskaya Nick Winter Miguel Orbegozo Rodriguez Robert Lauff Dustin Wehr Colin Tang Zaki Hossain Shaun Phillips Fortuna Samuele Fredrik Ekström Angela Hammon Oam Patel Faraz Farhidi George Medley Forough Mohammadzadeh Madellene Peñaflor Haile Kassahun Alena Friedrich Rayner Hernandez Perez Daniel Pyda Taom Sakal Omkar Dhamane Ali Khajegili Mirabadi Eric Hallman Kenchi Okutsu Mike Battaglia Mohammad Maghsoudimehrabani Alon Amit Dave Hulbert Roberto Pereira Simon Weber Handoko Anton Peristyy Stephen Malina Mustafa Mehkary Rami Aly Frank Reidegeld Anna-Katharina Dick Cary Friday Mukhwinder Singh Hassan Shapourian Wanyoung Kim Mariana Costa Hubeyb Gurdogan Harsh Kumar Chiara Ceconello Chao Zhuang Haon Park Micah Carroll Andrew R. Tawfeek Stefan Steinerberger Daattavya Aggarwal Michael Kirchhof Linjie Dai Evan Kim Johan Ferret Jainam Shah Yuzhou Wang Minghao Yan Krzysztof Burdzy Lixin Zhang Antonio Franca Diana T. Pham Kang Yong Loh Joshua Robinson Abram Jackson Paolo Giordano Philipp Petersen Adrian Cosma Jesus Colino Colin White Jacob Votava Vladimir Vinnikov Ethan Delaney Petr Spelda Vit Stritecky Syed M. Shahid Jean-Christophe Mourrat Lavr Vetoshkin Koen Sponselee Renas Bacho Zheng-Xin Yong Florencia de la Rosa Nathan Cho Xiuyu Li Guillaume Malod Orion Weller Guglielmo Albani Leon Lang Julien Laurendeau Dmitry Kazakov Fatimah Adesanya Julien Portier Lawrence Hollom Victor Souza Yuchen Anna Zhou Julien Degorre Yiğit Yalın Gbenga Daniel Obikoya Rai Filippo Bigi M. C. Boscá Oleg Shumar Kaniuar Bacho Gabriel Recchia Mara Popescu Nikita Shulga Ngefor Mildred Tanwie Thomas C. H. Lux Ben Rank Colin Ni Matthew Brooks Alesia Yakimchyk Huanxu Liu Stefano Cavalleri Olle Häggström Emil Verkama Joshua Newbould Hans Gundlach Leonor Brito-Santana Brian Amaro Vivek Vajipey Rynaa Grover Ting Wang Yosi Kratish Wen-Ding Li Sivakanth Gopi Andrea Caciolai Christian Schroeder de Witt Pablo Hernández-Cámara Emanuele Rodolà Jules Robins Dominic Williamson Vincent Cheng Brad Raynor Hao Qi Ben Segev Jingxuan Fan Sarah Martinson Erik Y. Wang Kaylie Hausknecht Michael P. Brenner Mao Mao Christoph Demian Peyman Kassani Xinyu Zhang David Avagian Eshawn Jessica Scipio Alon Ragoler Justin Tan Blake Sims Rebeka Plecnik Aaron Kirtland Omer Faruk Bodur D. P. Shinde Yan Carlos Leyva Labrador Zahra Adoul Mohamed Zekry Ali Karakoc Tania C. B. Santos Samir Shamseldeen Loukmane Karim Anna Liakhovitskaia Nate Resman Nicholas Farina Juan Carlos Gonzalez Gabe Maayan Earth Anderson Rodrigo De Oliveira Pena Elizabeth Kelley Hodjat Mariji Rasoul Pouriamanesh Wentao Wu Ross Finocchio Ismail Alarab Joshua Cole Danyelle Ferreira Bryan Johnson Mohammad Safdari Liangti Dai Siriphan Arthornthurasuk Isaac C. McAlister Alejandro José Moyano Alexey Pronin Jing Fan Angel Ramirez-Trinidad Yana Malysheva Daphiny Pottmaier Omid Taheri Stanley Stepanic Samuel Perry Luke Askew Raúl Adrián Huerta Rodríguez Ali M. R. Minissi Ricardo Lorena Krishnamurthy Iyer Arshad Anil Fasiludeen Ronald Clark Josh Ducey Matheus Piza Maja Somrak Eric Vergo Juehang Qin Benjámin Borbás Eric Chu Jack Lindsey Antoine Jallon I. M. J. McInnis Evan Chen Avi Semler Luk Gloor Tej Shah Marc Carauleanu Pascal Lauer Tran Đuc Huy Hossein Shahrtash Emilien Duc Lukas Lewark Assaf Brown Samuel Albanie Brian Weber Warren S. Vaz Pierre Clavier Yiyang Fan Gabriel Poesia Reis e Silva Long Lian Marcus Abramovitch Xi Jiang Sandra Mendoza Murat Islam Juan Gonzalez Vasilios Mavroudis Justin Xu Pawan Kumar Laxman Prasad Goswami Daniel Bugas Nasser Heydari Ferenc Jeanplong Thorben Jansen Antonella Pinto Archimedes Apronti Abdallah Galal Ng Ze-An Ankit Singh Tong Jiang Joan of Arc Xavier Kanu Priya Agarwal Mohammed Berkani Gang Zhang Zhehang Du Benedito Alves de Oliveira Junior Dmitry Malishev Nicolas Remy Taylor D. Hartman Tim Tarver Stephen Mensah Gautier Abou Loume Wiktor Morak Farzad Habibi Sarah Hoback Will Cai Javier Gimenez Roselynn Grace Montecillo Jakub Łucki Russell Campbell Asankhaya Sharma Khalida Meer Shreen Gul Daniel Espinosa Gonzalez Xavier Alapont Alex Hoover Gunjan Chhablani Freddie Vargus Arunim Agarwal Yibo Jiang Deepakkumar Patil David Outevsky Kevin Joseph Scaria Rajat Maheshwari Abdelkader Dendane Priti Shukla Ashley Cartwright Sergei Bogdanov Niels Mündler Sören Möller Luca Arnaboldi Kunvar Thaman Muhammad Rehan Siddiqi Prajvi Saxena Himanshu Gupta Tony Fruhauff Glen Sherman Mátyás Vincze Siranut Usawasutsakorn Dylan Ler Anil Radhakrishnan Innocent Enyekwe Sk Md Salauddin Jiang Muzhen Aleksandr Maksapetyan Vivien Rossbach Chris Harjadi Mohsen Bahaloohoreh Claire Sparrow Jasdeep Sidhu Sam Ali Song Bian John Lai Eric Singer Justine Leon Uro Greg Bateman Mohamed Sayed Ahmed Menshawy Darling Duclosel Dario Bezzi Yashaswini Jain Ashley Aaron Murat Tiryakioglu Sheeshram Siddh Keith Krenek Imad Ali Shah Jun Jin Scott Creighton Denis Peskoff Zienab EL-Wasif Ragavendran P Michael Richmond Joseph McGowan Tejal Patwardhan Hao-Yu Sun Ting Sun Nikola Zubić Samuele Sala Stephen Ebert Jean Kaddour Manuel Schottdorf Dianzhuo Wang Gerol Petruzella Alex Meiburg Tilen Medved Ali ElSheikh S Ashwin Hebbar Lorenzo Vaquero Xianjun Yang Jason Poulos Vilém Zouhar Sergey Bogdanik Mingfang Zhang Jorge Sanz-Ros David Anugraha Yinwei Dai Anh N. Nhu Xue Wang Ali Anil Demircali Zhibai Jia Yuyin Zhou Juncheng Wu Mike He Nitin Chandok Aarush Sinha Gaoxiang Luo Long Le Mickaël Noyé Michał Perełkiewicz Ioannis Pantidis Tianbo Qi Soham Sachin Purohit Letitia Parcalabescu Thai-Hoa Nguyen Genta Indra Winata Edoardo M. Ponti Hanchen Li Kaustubh Dhole Jongee Park Dario Abbondanza Yuanli Wang Anupam Nayak Diogo M. Caetano Antonio A. W. L. Wong Maria del Rio-Chanona Dániel Kondor Pieter Francois Ed Chalstrey Jakob Zsambok Dan Hoyer Jenny Reddish Jakob Hauser Francisco-Javier Rodrigo-Ginés Suchandra Datta Maxwell Shepherd Thom Kamphuis Qizheng Zhang Hyunjun Kim Ruiji Sun Jianzhu Yao Franck Dernoncourt Satyapriya Krishna Sina Rismanchian Bonan Pu Francesco Pinto Yingheng Wang Kumar Shridhar Kalon J. Overholt Glib Briia Hieu Nguyen David Soler Bartomeu Tony CY Pang Adam Wecker Yifan Xiong Fanfei Li Lukas S. Huber Joshua Jaeger Romano De Maddalena Xing Han Lù Yuhui Zhang Claas Beger Patrick Tser Jern Kon Sean Li Vivek Sanker Ming Yin Yihao Liang Xinlu Zhang Ankit Agrawal Li S. Yifei Zechen Zhang Mu Cai Yasin Sonmez Costin Cozianu Changhao Li Alex Slen Shoubin Yu Hyun Kyu Park Gabriele Sarti Marcin Briański Alessandro Stolfo Truong An Nguyen Mike Zhang Yotam Perlitz Jose Hernandez-Orallo Runjia Li Amin Shabani Felix Juefei-Xu Shikhar Dhingra Orr Zohar My Chiffon Nguyen Alexander Pondaven Abdurrahim Yilmaz Xuandong Zhao Chuanyang Jin Muyan Jiang Stefan Todoran Xinyao Han Jules Kreuer Brian Rabern Anna Plassart Martino Maggetti Luther Yap Robert Geirhos Jonathon Kean Dingsu Wang Sina Mollaei Chenkai Sun Yifan Yin Shiqi Wang Rui Li Yaowen Chang Anjiang Wei Alice Bizeul Xiaohan Wang Alexandre Oliveira Arrais Kushin Mukherjee Jorge Chamorro-Padial Jiachen Liu Xingyu Qu Junyi Guan Adam Bouyamourn Shuyu Wu Martyna Plomecka Junda Chen Mengze Tang Jiaqi Deng Shreyas Subramanian Haocheng Xi Haoxuan Chen Weizhi Zhang Yinuo Ren Haoqin Tu Sejong Kim Yushun Chen Sara Vera Marjanović Junwoo Ha Grzegorz Luczyna Jeff J. Ma Zewen Shen Dawn Song Cedegao E. Zhang Zhun Wang Gaël Gendron Yunze Xiao Leo Smucker Erica Weng Kwok Hao Lee Zhe Ye Stefano Ermon Ignacio D. Lopez-Miguel Theo Knights Anthony Gitter Namkyu Park Boyi Wei Hongzheng Chen Kunal Pai Ahmed Elkhanany Han Lin Philipp D. Siedler Jichao Fang Ritwik Mishra Károly Zsolnai-Fehér Xilin Jiang Shadab Khan Jun Yuan Rishab Kumar Jain Xi Lin Mike Peterson Zhe Wang Aditya Malusare Maosen Tang Isha Gupta Ivan Fosin Timothy Kang Barbara Dworakowska Kazuki Matsumoto Guangyao Zheng Gerben Sewuster Jorge Pretel Villanueva Ivan Rannev Igor Chernyavsky Jiale Chen Deepayan Banik Ben Racz Wenchao Dong Jianxin Wang Laila Bashmal Duarte V. Gonçalves Wei Hu Kaushik Bar Ondrej Bohdal Atharv Singh Patlan Shehzaad Dhuliawala Caroline Geirhos Julien Wist Yuval Kansal Bingsen Chen Kutay Tire Atak Talay Yücel Brandon Christof Veerupaksh Singla Zijian Song Sanxing Chen Jiaxin Ge Kaustubh Ponkshe Isaac Park Tianneng Shi Martin Q. Ma Joshua Mak Sherwin Lai Antoine Moulin Zhuo Cheng Zhanda Zhu Ziyi Zhang Vaidehi Patil Ketan Jha Qiutong Men Jiaxuan Wu Tianchi Zhang Bruno Hebling Vieira Alham Fikri Aji Jae-Won Chung Mohammed Mahfoud Ha Thi Hoang Marc Sperzel Wei Hao Kristof Meding Sihan Xu Vassilis Kostakos Davide Manini Yueying Liu Christopher Toukmaji Jay Paek Eunmi Yu Arif Engin Demircali Zhiyi Sun Ivan Dewerpe Hongsen Qin Roman Pflugfelder James Bailey Johnathan Morris Ville Heilala Sybille Rosset Zishun Yu Peter E. Chen Woongyeong Yeo Eeshaan Jain Ryan Yang Sreekar Chigurupati Julia Chernyavsky Sai Prajwal Reddy Subhashini Venugopalan Hunar Batra Core Francisco Park Hieu Tran Guilherme Maximiano Genghan Zhang Yizhuo Liang Hu Shiyu Rongwu Xu Rui Pan Siddharth Suresh Ziqi Liu Samaksh Gulati Songyang Zhang Peter Turchin Christopher W. Bartlett Christopher R. Scotese Phuong M. Cao Ben Wu Jacek Karwowski Davide Scaramuzza Aakaash Nattanmai Gordon McKellips Anish Cheraku Asim Suhail Ethan Luo Marvin Deng Jason Luo Ashley Zhang Kavin Jindel Jay Paek Kasper Halevy Allen Baranov Michael Liu Advaith Avadhanam David Zhang Vincent Cheng Brad Ma Evan Fu Liam Do Joshua Lass Hubert Yang Surya Sunkari Vishruth Bharath Violet Ai James Leung Rishit Agrawal Alan Zhou Kevin Chen Tejas Kalpathi Ziqi Xu Gavin Wang Tyler Xiao Erik Maung Sam Lee Ryan Yang Roy Yue Ben Zhao Julia Yoon Sunny Sun Aryan Singh Ethan Luo Clark Peng Tyler Osbey Taozhi Wang Daryl Echeazu Hubert Yang Timothy Wu Spandan Patel Vidhi Kulkarni Vijaykaarti Sundarapandiyan Ashley Zhang Andrew Le Zafir Nasim Srikar Yalam Ritesh Kasamsetty Soham Samal Hubert Yang David Sun Nihar Shah Abhijeet Saha Alex Zhang Leon Nguyen Laasya Nagumalli Kaixin Wang Alan Zhou Aidan Wu Jason Luo Anwith Telluri Steven Dillmann Zhengxiang Wang Junyu Luo Hugo Lunn Artem Gazizov Haitz Sáez de Ocáriz Borde Ivan Trus Morgan Hervault Zheyu Zhang Bo Chen Yuchen Wu Christopher J. Cordier Gün Kaynar Cansin Ayvaz Polina Avdiunina Johannes Brust Xingjian Diao K. D. Meaney Yifan Gu Chenyu Wang Chenzhuo Dong William Wright Simon Brave Owen Root Jiayuan Liu Chow Chun Lok Tianqin Li Shiyi Du Dailan He Lufeiya Liu Sina Jamalzadegan Anil Ramakrishna Xuanqing Xu Xin Qing Xin Luo Wenkai Li Shi Bo Filipp Gusev Maximos Skandalis Desheng Ma Chunhui Zhang Haoran Qiu Allen G Hart Rickard Brüel Gabrielsson Ido Akov Artem Lukoianov Summer Yue Alexandr Wang Dan Hendrycks
多模态 学术评测 大模型 知识边界 自动评分

核心发现

方法论

HLE采用多模态题库,涵盖数学、科学、文史等领域,题目由全球专家设计,包含选择题和简答题,均具备明确解答。模型评估采用自动评分系统,结合多轮推理和知识验证机制,确保题目难度超越现有基准。模型性能以准确率和校准度衡量,特别关注模型在高难度题目的表现。采用特定的推理路径分析和知识检索机制,评估模型的理解深度和推理能力。

关键结果

  • 在2500题中,最先进的LLMs(如GPT-4)平均准确率仅为35%,远低于人类专家的90%以上。模型在数学、物理、历史等科目中的表现差异显著,特别是在需要复杂推理和跨领域知识融合时,表现尤为不足。模型校准度低,误差分布偏离理想状态,显示其对题意理解仍有限。
  • 模型在多模态题目中的表现明显低于纯文本任务,尤其在图表理解和多步推理环节,准确率下降至20%-30%。
  • 通过对模型不同架构(如GPT-4、PaLM 2)进行对比,发现模型能力提升有限,表明当前技术尚难突破人类专家水平,特别是在长尾知识和复杂推理方面。

研究意义

HLE作为首个覆盖广泛学科、采用多模态题目的终极学术基准,揭示了当前大模型在高阶认知任务中的巨大差距。它为学界提供了一个衡量模型极限的工具,有助于推动模型理解和推理能力的突破,也为AI在教育、科研等领域的应用设定了新标杆。该基准强调模型在知识深度和推理复杂性上的不足,促使研究者关注模型的知识整合和推理机制的改进。

技术贡献

本文提出了多模态、跨学科的学术评测框架,结合自动化评分系统,创新性地设计了难度超越现有基准的题库。引入多轮推理和知识验证机制,增强模型的理解深度。通过系统性评估不同模型架构,验证了模型在复杂推理任务中的局限性,为未来模型设计提供了理论指导。该工作还建立了一个开放的评测平台,促进学界对模型能力的持续追踪。

新颖性

首次构建涵盖2500题、多模态、多学科的终极学术基准,强调模型在高阶认知和跨领域推理中的表现。不同于传统单一任务或文本基准,HLE引入多模态题目,结合全球专家设计,确保题目难度和多样性,开创了学术评测的新范式。

局限性

  • 当前模型在多模态理解和长尾知识推理方面仍表现不足,难以应对极端复杂题目,反映模型在知识深度和推理能力上的瓶颈。
  • 评测依赖自动评分系统,可能在某些主观题或开放性问题上存在偏差,限制了全面评估。
  • 题目设计虽由专家把关,但仍可能存在偏向某些学科或知识体系的风险,未来需多元化题库来源。

未来方向

未来将扩展题库规模,增加开放式题目,探索多模态交互机制。加强模型在跨学科推理、知识整合和少样本学习中的能力,推动模型向人类专家水平迈进。此外,将结合人类评估与自动化手段,完善模型校准和解释能力,促进模型在实际应用中的可靠性和透明度。

AI 总览摘要

在人工智能快速发展的背景下,评测体系的难度不断追赶模型能力,传统基准如MMLU已无法区分最先进模型的细微差异。本文提出了‘Humanity's Last Exam’(HLE),这是一个涵盖2500题、多模态、多学科的终极学术评测平台,旨在突破现有模型的瓶颈。由全球学科专家设计,题目涉及数学、科学、历史、文史等领域,结合选择题和简答题,确保题目具有明确、可验证的解答,同时难度超越互联网检索能力。模型评估采用自动评分系统,结合多轮推理和知识验证机制,全面衡量模型的理解深度和推理能力。实验证明,当前最先进的模型如GPT-4在HLE上的平均准确率仅为35%,远低于人类专家的90%以上,显示出巨大差距。这一结果强调了模型在复杂认知任务中的局限性,揭示了AI在高阶学术推理中的潜在瓶颈。HLE的推出不仅为学术界提供了一个极限测试平台,也为未来模型设计指明了方向。它促使研究者关注模型的知识深度、跨模态理解和推理机制的改进,推动AI迈向更接近人类认知的水平。未来工作将扩展题库规模,增加开放性题目,结合多模态交互,持续推动模型能力的突破。总体而言,HLE代表了AI能力评估的一个里程碑,开启了模型极限探索的新篇章。

深度分析

研究背景

随着大规模预训练模型(如GPT系列、PaLM系列)的出现,AI在自然语言理解和生成方面取得了显著突破。然而,现有评测基准如MMLU、BIG-Bench等,已难以区分最先进模型的细微差异,主要因为模型在大部分任务中已达到或超过90%的准确率。尽管如此,这些基准多为单一模态、偏向事实记忆或简单推理,难以反映模型在复杂学术推理中的真实能力。近年来,学界开始关注多模态、多学科的评测体系,试图突破模型的认知边界,但缺乏统一、全面的终极基准。由此,提出了‘Humanity’s Last Exam’(HLE),旨在填补这一空白,推动模型在高阶认知和多模态理解上的突破。

核心问题

当前大模型在学术推理、跨学科知识融合及复杂推理任务中表现不足,尤其在多模态信息处理和长尾知识推理方面存在明显短板。传统基准无法充分反映模型在真实学术场景中的能力,限制了模型的实际应用潜力。面对人类专家的高标准,模型仍难以胜任高难度题目,亟需更具挑战性和多样性的评测体系,以全面衡量模型的理解深度和推理能力。

核心创新

本研究的核心创新在于:1)构建全球专家设计的2500题多模态学术题库,覆盖广泛学科;2)引入多轮推理和知识验证机制,提升模型推理能力;3)采用自动化评分系统,确保评测效率和客观性;4)强调题目难度超越互联网检索,考验模型的深层理解。相比传统单一任务基准,HLE更贴近人类学术能力的极限,推动模型在跨模态、多学科推理中的突破。

方法详解

  • �� 题库设计:由全球学科专家团队设计,确保题目多样性和难度。• 多模态题目:结合文本、图表、图片等信息,测试模型的多模态理解能力。• 自动评分:利用规则和模型结合的自动化系统,确保评分的客观性和一致性。• 多轮推理:设计题目需模型进行多步推理,验证其推理路径的合理性。• 知识验证:引入知识检索机制,确保模型在推理中调用正确的知识点。• 评估指标:采用准确率、校准度和推理路径合理性等多维指标。• 题目难度:设计难度超越普通检索,要求模型具备深层理解和跨学科融合能力。

实验设计

  • �� 数据集:2500题,涵盖数学、物理、历史、文学等学科。• 模型:包括GPT-4、PaLM 2、Claude等。• 评估:准确率、校准度、推理路径合理性。• 超参数:模型温度设定为0.7,推理轮数为3-5轮。• 比较:与传统基准(如MMLU)对比,分析模型在不同学科和题型上的表现差异。• Ablation:逐步去除多模态输入、多轮推理机制,验证各模块贡献。

结果分析

  • �� GPT-4在HLE上的平均准确率为35%,远低于人类专家的90%以上。• 在图表理解和跨学科推理题中,准确率下降至20%-30%。• 模型校准度低,表现出明显的偏差和不确定性。• 不同模型架构表现差异明显,表明模型能力提升空间巨大。• 结果显示,当前模型在高阶学术推理中仍有明显短板,亟待技术突破。

应用场景

  • �� 教育:可作为高等学府的学术能力评估工具,推动个性化教学。• 科研:辅助科研人员进行跨学科知识整合和复杂推理。• AI研发:指导模型架构优化,突破认知瓶颈。• 政策:为AI能力监管提供科学依据,制定未来发展战略。

局限与展望

  • �� 题库虽由专家设计,但仍可能存在偏向某些学科或知识体系的风险。• 自动评分系统在开放性题目上可能偏差较大,影响评估全面性。• 模型在极端复杂推理和长尾知识方面仍表现不足,限制其应用范围。• 未来需结合人类评审,完善评测机制,提升公平性和准确性。

通俗解读 非专业人士也能看懂

想象你在参加一场超级难的知识考试,这个考试题目不仅涉及数学、历史、科学,还包括图片和图表。每题都像是一个谜题,只有真正理解了题意,才能找到正确答案。现在的AI就像一个学生,虽然能答一些简单题,但面对这些复杂、跨学科的题目时,却像迷路的小孩。这个‘Humanity’s Last Exam’就像是一个终极考场,测试AI是否真的理解了这些知识。它设计了2500个难题,确保只有真正聪明的学生才能答对。通过这个考试,我们可以知道AI在哪些方面还需要努力,比如理解复杂图表、跨领域推理和长尾知识。这个平台就像一面镜子,照出AI的真实水平,也推动AI变得更聪明、更像人类。未来,AI如果能通过这个考试,就意味着它已经迈向了真正的智能时代。

简单解释 像给14岁少年讲一样

想象你在参加一场超级难的考试,题目不仅有数学题、历史题,还会有图片和图表。你得用脑子把这些信息结合起来,找到答案。现在的AI就像一个学生,能答一些简单题,但面对这些复杂题时,就像迷路一样。这个‘Humanity’s Last Exam’就像是一个终极考场,专门用来测试AI到底懂不懂这些知识。它设计了2500个难题,确保只有真正聪明的学生才能答对。通过这个考试,我们可以知道AI在哪些方面还不够聪明,比如理解复杂的图表、跨学科的推理和记忆那些不常见的知识。这就像一面镜子,照出AI的真实水平,也让我们知道未来还需要在哪些方面努力。要是AI能通过这个考试,就意味着它变得更聪明、更像人类了。这个平台会推动AI变得更强大,帮助我们创造出更聪明、更有用的机器人。

原文摘要

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve over 90\% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities. In response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a significant gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

cs.LG cs.AI cs.CL