OmniScientist: An Omni-Modal Omni-Discipline AI Scientist
OmniScientist employs multimodal perception and a multi-agent architecture to automate multidisciplinary research from raw evidence to publication, achieving an average score of 6.3.
Key Findings
Methodology
The core of OmniScientist integrates an end-to-end multimodal perception layer with three autonomous agents—ideation, experimentation, and report generation—operating within a deterministic pipeline. The perception layer directly processes heterogeneous raw evidence, including images, signals, videos, 3D structures, tables, formulas, and graphs, preserving spatial, temporal, cross-channel, and procedural relations. Each agent employs a ReAct (Reasoning + Action) loop, enabling dynamic interaction and evidence-driven decision-making. Code-enforced checks—such as novelty screening, statistical validity, provenance tracking, and numerical traceability—ensure scientific rigor. The system was evaluated on 36 real-world cases spanning five disciplines, four evidence families, and multiple modalities, successfully completing the entire research cycle from raw data to a published paper with a mean score of 6.3, outperforming variants with only precomputed features.
Key Results
- The system successfully completed the full research cycle in 36 real datasets, achieving an average paper score of 6.3 out of 7, covering diverse modalities such as images, signals, videos, 3D structures, tables, formulas, and graphs.
- Compared to a baseline receiving only precomputed scalar features, the full perception system improved all seven evaluation dimensions, with an 85% win rate in head-to-head judgments, especially excelling in multimodal grounding and scientific significance.
- Ablation studies confirmed that removing raw perception led to significant drops in quality, highlighting the importance of direct evidence processing for hypothesis formulation and conclusion support.
Significance
This work addresses a fundamental limitation in existing AI scientific systems, which rely heavily on preprocessed summaries, losing critical spatial, temporal, and procedural information. By maintaining the integrity of raw evidence throughout the research lifecycle, OmniScientist enhances the depth, reliability, and interpretability of automated scientific discovery. Its architecture demonstrates that lifecycle-wide perception is essential for evidence-grounded reasoning, paving the way for AI to become a more trustworthy and autonomous partner in scientific research across disciplines. This advancement has profound implications for accelerating innovation in fields like materials science, medicine, and environmental science, where complex multimodal data are prevalent.
Technical Contribution
The paper introduces a novel multi-agent, perception-first framework that integrates raw multimodal evidence into a deterministic pipeline, with each stage enforced by code-based validation. Key innovations include: • A hierarchical perception layer that directly interprets heterogeneous data, preserving spatial, temporal, cross-channel, and procedural relations; • Multi-agent coordination via ReAct loops, enabling dynamic hypothesis generation, experimental planning, and reporting; • Rigorous code-based checks for novelty, statistical validity, provenance, and traceability at each stage; • A comprehensive validation suite across five disciplines and four evidence families, demonstrating broad applicability. These contributions collectively push the frontier of autonomous scientific discovery, enabling end-to-end, evidence-grounded research with high reliability and interpretability.
Novelty
This study is the first to realize a fully end-to-end, lifecycle-spanning scientific AI system that directly perceives raw, heterogeneous evidence across multiple disciplines. Unlike prior work limited to fixed features or isolated perception modules, OmniScientist maintains the integrity of spatial, temporal, and procedural relations throughout the research process. Its multi-agent, code-verified pipeline ensures that every stage—from hypothesis to conclusion—is grounded in authentic evidence, significantly advancing the state of the art in autonomous scientific reasoning and multi-modal data integration.
Limitations
- The system relies on high-quality, well-annotated raw data, which can be costly and challenging to obtain in some domains, potentially limiting scalability.
- While robust in controlled environments, its performance under noisy, ambiguous, or incomplete evidence scenarios requires further validation.
- Current implementation is computationally intensive, especially for high-dimensional multimodal data, necessitating hardware acceleration and optimization for real-time applications.
Future Work
Future research will focus on enhancing robustness against noisy data, integrating more sophisticated reasoning modules, and expanding the system’s capabilities to handle larger, more complex datasets. Additionally, efforts will be made to streamline computational efficiency, facilitate real-world deployment, and develop human-AI collaborative interfaces, ultimately aiming for fully autonomous, scalable scientific discovery platforms.
AI Executive Summary
In the realm of scientific research, the complexity and diversity of evidence have long posed challenges for automation. Traditional AI systems predominantly process preprocessed summaries or features, which often strip away critical spatial, temporal, and procedural information embedded in raw data. This limitation hampers the system’s ability to generate meaningful hypotheses, design experiments, and support conclusions with robust evidence.
OmniScientist addresses this gap by introducing a comprehensive, end-to-end framework that directly perceives heterogeneous raw evidence across multiple disciplines. Its architecture comprises a hierarchical perception layer and three autonomous agents—ideation, experimentation, and report generation—working within a deterministic pipeline. The perception layer employs advanced tools to interpret images, signals, videos, 3D structures, tables, formulas, and graphs, preserving their intrinsic relations. This raw evidence feeds into the agents, which utilize the ReAct (Reasoning + Action) paradigm to iteratively formulate hypotheses, design and execute experiments, analyze results, and compile scientific manuscripts.
Crucially, the system enforces rigorous code-based checks at each stage, including novelty screening, statistical validation, provenance tracking, and numerical traceability. These mechanisms ensure that the entire research process adheres to scientific standards, minimizes biases, and maintains transparency. The evaluation on 36 real datasets spanning physics, astronomy, biology, agriculture, and engineering demonstrated the system’s ability to produce high-quality, evidence-grounded papers with an average score of 6.3, outperforming variants limited to precomputed features.
The results underscore that lifecycle-wide perception is vital for evidence-grounded scientific discovery. By maintaining the integrity of raw multimodal data, OmniScientist significantly enhances the depth, reliability, and interpretability of automated research. Its broad applicability across disciplines and evidence types indicates a promising future for AI-driven scientific exploration. Moving forward, efforts will focus on improving robustness, computational efficiency, and real-world deployment, aiming to transform AI into a fully autonomous partner in scientific innovation.
Deep Analysis
Background
科学研究的历史经历了从符号推理到多模态信息融合的演变。早期的AI科研多依赖逻辑推理和知识图谱（如Cycorp的Cyc系统），解决结构化数据处理问题。随着深度学习的崛起，图像识别（如ImageNet）、信号分析（如Spectrogram）、自然语言处理（如BERT、GPT）等技术取得突破，推动多模态模型（如CLIP、VisualBERT）在科学领域的应用。近年来，端到端自动化科研系统逐步出现（如Lu et al., 2026），能够提出假设、设计实验、撰写论文，但多依赖预定义特征，缺乏对原始多模态证据的全面感知。现有系统在保持空间、时间和程序关系方面存在明显不足，限制了深度推理和跨学科融合的能力。
Core Problem
核心难题在于，现有AI科研系统多依赖预处理或预定义特征，无法充分利用原始多模态证据中的空间、时间和程序关系，导致科学推理深度不足。缺乏对证据的直接感知，限制了问题提出、假设生成和结论支持的能力。如何实现从原始多模态数据到科学论文的全流程自动化，保持证据的完整关系，成为亟待突破的关键。解决此问题不仅关系到科研自动化的可信度，也影响跨学科、多模态信息的深度融合与推理能力。
Innovation
本研究的创新点包括：
- �� 引入多模态感知层，直接识别和保持原始证据中的空间、时间和程序关系，避免信息在接口中丢失；
- �� 设计多智能体（思想、实验、写作）在确定性管道中协作，利用ReAct循环实现动态交互，增强自主推理能力；
- �� 在每个阶段引入代码检验机制，确保科研的严谨性，包括新颖性筛查、统计有效性、执行溯源和数值追溯；
- �� 构建跨学科、多模态的真实案例验证体系，涵盖图像、信号、视频、三维结构、表格、公式和图形，验证系统的广泛适用性。这些创新突破了传统模型对预定义特征的依赖，推动了端到端、多模态、多智能体科研自动化的发展。
Methodology
- �� 感知层：利用多模态感知工具识别原始证据（图像、信号、视频、三维模型、表格、公式、图形），保持空间、时间和程序关系，提供丰富基础信息。
- �� 智能体设计：三个智能体（思想生成、实验执行、论文写作）在确定性管道中协作，采用ReAct（推理+行动）循环，不断交互调整研究策略。
- �� 代码检验：在每个阶段引入自动验证机制，确保新颖性（筛查重复和HARKing）、统计有效性（多重比较校正）、执行溯源（实验记录追溯）和数值可追溯性。
- �� 跨学科验证：构建涵盖五大学科类别、四类证据类型的36个真实案例，验证系统的通用性和性能表现。
- �� 任务定义：输入为数据集、研究主题和目标属性，系统自主提出假设、设计实验、分析结果、撰写论文，无需人工干预。
Experiments
- �� 数据集：涵盖物理、天文、生命医学、农业、工程等五大学科类别，使用真实公开数据（如NFFA-EUROPE、GZ DECaLS、Kather CRC等），涉及多模态（图像、信号、视频、三维、表格、公式、图形）数据。
- �� 评估指标：论文整体评分（满分7分）、七个维度的详细评分（科学性、创新性、严谨性等），以及头对头判断的胜率（85%）。
- �� 实验设置：对比完整感知系统与只使用预计算特征的版本，进行AB测试，分析不同模态和阶段的贡献。
- �� 超参数：感知工具调用频率、智能体交互轮次、检验机制阈值等，均通过交叉验证优化。
- �� 结果验证：通过多案例验证系统的跨学科适应性和性能稳定性，进行消融实验分析感知对科研质量的影响。
Results
- �� 完成36个真实案例的全流程，平均论文得分6.3，显著优于只用预计算特征的版本（得分约5.2），验证了感知在科学发现中的关键作用。
- �� 头对头比较中，感知系统赢得85%的判断，特别在多模态基础和科学意义两个维度表现突出，表明原始证据的完整感知极大提升了科研质量。
- �� 消融实验显示，去除感知模块后，论文质量下降明显，特别在空间关系和程序关系的保持方面，影响了假设提出和结论支持的深度。
Applications
- �� 立即应用：该系统可用于基础科学研究、医学诊断、材料设计等领域，自动分析多模态数据，提出科学假设，设计验证实验，自动撰写科研论文，极大提高科研效率和可信度。
- �� 长期愿景：未来将实现全流程自动化的科学发现流程，推动AI成为科研的核心工具，辅助科学家发现未知规律，推动新材料、新药、环境监测等关键技术的突破。
Plain Language Accessible to non-experts
想象你在一个大厨房里做菜，里面有各种各样的食材、调料、厨具和烹饪设备。这些东西会产生不同的原始信息，比如食材的图片、味道的信号、烹饪的视频、食材的三维模型和配料的公式。以前，厨师们只会用这些信息的总结，比如菜谱、配料表或者简单的描述，来做菜。这样会丢失很多细节，比如食材的具体结构、烹饪的动态变化、步骤的顺序。
现在，想象有一个超级厨师机器人，它可以直接“看”这些原始的食材和信息，就像我们用眼睛观察一样。这个机器人有三个助手：一个负责提出问题（比如“这个菜为什么会变得特别香？”），一个负责设计和做实验（比如“用不同的火候试试”），还有一个负责写菜谱（总结所有的步骤和结果）。
这个机器人不仅能理解各种不同类型的原始信息，还能在每一步都确保自己没有犯错误，比如确认烹饪的步骤是合理的、结果是真实的，没有被噪声误导。它会不断检查自己的工作，确保每个结论都可以追溯到原始的食材和过程。
最终，这个超级厨师可以从一堆复杂的原始信息开始，自动写出一份完整的菜谱，里面详细描述每个步骤和发现。这意味着未来的厨房会变得更智能、更高效，甚至可以帮人们发现新的菜肴、新的调料组合，解决各种烹饪难题。
Abstract
Recent advances in foundation models have enabled AI scientists to automate increasingly complete research workflows, from hypothesis generation and code execution to manuscript preparation. Yet workflow coverage alone does not provide access to the full evidence on which scientific discovery depends. Existing systems typically reason over text, code, labels, or precomputed summaries, leaving scientifically decisive spatial, temporal, cross-channel, and procedural relations unavailable to the agent. We introduce OmniScientist, an end-to-end, omni-modal AI scientist that conducts multidisciplinary research directly from heterogeneous raw evidence. A perception layer and 3 autonomous agents for ideation, experiment, and writeup operate within a deterministic pipeline, allowing observations to shape research questions, experimental decisions, and final claims throughout the research lifecycle. By running idea, rigour, and claim checks in code, the system enforces novelty screening, statistical validity, execution provenance, and numerical traceability. We evaluate OmniScientist on 36 real-data cases spanning 5 discipline families, 4 families of scientific evidence, and modalities including images, signals, audio, video, 3-D structures, trajectories, tables, formulae, and graphs. The system completes the full path from raw data to a compiled manuscript in all 36 cases and achieves a mean overall paper score of 6.3 with the reference reasoning backbone. In paired comparisons against a blind variant that receives only precomputed scalar features, direct perception improves all 7 evaluation dimensions and wins 85% of head-to-head judgments. These results show that lifecycle-wide perception is essential for evidence-grounded scientific discovery and provides a practical path toward broadly capable AI scientists.
References (20)
The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
Yutaro Yamada, R. Lange, Cong Lu et al.
Agent Laboratory: Using LLM Agents as Research Assistants
Samuel Schmidgall, Yusheng Su, Ze Wang et al.
MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
Zekun Li, Xianjun Yang, Kyuri Choi et al.
Evaluating large language model agents for automation of atomic force microscopy
I. Mandal, J. Soni, Mohd Zaki et al.
FORTIS: Benchmarking Over-Privilege in Agent Skills
Shawn Li, Chenxiao Yu, Hanqi Wang et al.
PubChem 2025 update.
Sunghwan Kim, Jie Chen, Tiejun Cheng et al.
AI-Researcher: Autonomous Scientific Innovation
Jiabin Tang, Lianghao Xia, Zhonghang Li et al.
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe et al.
An Objective Comparison of Cell Tracking Algorithms
V. Ulman, Martin Maška, Klas E. G. Magnusson et al.
ORGANA: A Robotic Assistant for Automated Chemistry Experimentation and Characterization
K. Darvish, Marta Skreta, Yuchi Zhao et al.
OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
Jason Priem, Heather A. Piwowar, Richard Orr
A Large-Scale Annotated Mechanical Components Benchmark for Classification and Retrieval Tasks with Deep Neural Networks
Sangpil Kim, Hyung-gun Chi, Xiao Hu et al.
Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey
C. Lintott, K. Schawinski, A. Slosar et al.
μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Alejandro Lozano, Jeffrey J. Nirschl, James Burgess et al.
Visual Instruction Tuning
Haotian Liu, Chunyuan Li, Qingyang Wu et al.
MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification
Jiancheng Yang, Rui Shi, D. Wei et al.
PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
M. Takamoto, T. Praditia, Raphael Leiteritz et al.
A data-driven statistical model for predicting the critical temperature of a superconductor
K. Hamidieh
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Xiang Yue, Yuansheng Ni, Kai Zhang et al.
Towards end-to-end automation of AI research
Chris Lu, Cong Lu, R. Lange et al.