SciEGQA: A Dataset for Scientific Evidence-Grounded Question Answering and Reasoning
SciEGQA introduces semantic-region grounding and Grounding–Crop–then–Answer, with 1,623 human QA pairs and 30K+ training pairs.
Key Findings
Methodology
Each sample is formalized as S=(D,Q,A,T,E), where E contains page indices, bounding boxes (x1,y1,x2,y2), and semantic labels such as text, figure, or table. The benchmark covers SPSR, SPMR, and MPMR reasoning. Human construction used 12 annotators and 24 domain experts; automated construction uses SAM3 segmentation, VLM filtering/grouping, GPT-5.2 or Gemini 3 generation, and evidence-only verification.
Key Results
- The human benchmark contains 80 arXiv papers, 1,941 pages, and 1,623 QA pairs. SPSR, SPMR, and MPMR account for 46.15%, 34.26%, and 19.59%, respectively, providing a measured mixture of local, multi-region, and cross-page reasoning.
- The training set contains 3,671 papers, 42,380 pages, and 30,780 QA pairs. The paper reports that most existing VLMs obtain below 40% IoU for evidence grounding, demonstrating that page-level access does not imply precise evidence identification.
- Grounding–Crop–then–Answer uses Qwen3.5-27B as a fixed answerer after predicted regions are cropped, controlling for answer-generation differences. IoU@τ is measured at τ∈{0.3,0.5,0.7}; training on SciEGQA substantially improves reasoning, although the supplied excerpt omits model-by-model gains.
Significance
SciEGQA changes scientific DocVQA from answer-only evaluation to evidence-auditable reasoning. It addresses coarse page supervision, semantically fragmented token labels, and isolated figure/table tasks that ignore document context. The benchmark can therefore test whether a model actually uses the relevant paragraph, table, or figure. This is important for trustworthy literature assistants, searchable research archives, and industrial knowledge systems where unsupported answers are costly.
Technical Contribution
Its central technical contribution is semantic-region evidence representation: complete text blocks, tables, or figures receive bounding boxes, preserving semantic closure while enabling spatial and multimodal reasoning. The Grounding–Crop–then–Answer protocol predicts boxes, crops them, and invokes a shared Qwen3.5-27B answerer, separating localization failure from reasoning failure. For scalable training, SAM3 performs visual segmentation directly, avoiding dependence on fixed rule-based OCR/layout tools such as pdfplumber and preserving flexible visual units.
Novelty
Compared with DocVQA, MP-DocVQA, SciDQA, and PeerQA, which mainly provide page- or document-level evidence, and SPIQA, Charxiv, and AnaFig, which isolate tables, charts, or figures, SciEGQA combines semantic-region boxes, multimodal evidence, and multi-page scientific reasoning. Its fundamental novelty is the intermediate granularity: more precise than pages, yet more meaningful than token fragments.
Limitations
- The manually annotated benchmark contains only 80 papers and 1,623 questions, so coverage of disciplines, writing styles, and rare scientific layouts remains limited. The supplied text also does not provide full inter-annotator statistics or per-domain difficulty.
- The automatic set depends on SAM3, GPT-5.2, and Gemini 3, so segmentation, question-generation, and verification biases may propagate. Evidence-only answer verification checks answerability, not necessarily scientific factual truth.
- The provided excerpt ends before the complete evaluation suite; it lacks the full VLM list, baseline scores, training hyperparameters, and detailed ablations.
Future Work
Future work should broaden disciplines and document styles, add formula and chart verification, and audit automatically generated questions with experts. Full model-level results and ablations should compare segmentation and generator choices. Methodologically, promising directions include joint localization–reasoning training, long-context evidence retrieval, calibrated uncertainty, and factual validation against the scientific source.
AI Executive Summary
Scientific papers are not ordinary images. A single page may combine dense prose, tables, figures, equations, and cross-page references. Existing DocVQA benchmarks often identify only the relevant page: a model can produce a plausible answer without revealing which paragraph or visual element supports it. Token-level supervision is more precise but can fragment semantic units, while component datasets such as SPIQA, Charxiv, and AnaFig often isolate tables, charts, or figures from the surrounding argument.
SciEGQA introduces semantic-region evidence grounding. Each example records the document, question, answer, question type, and page-level bounding boxes for coherent evidence regions, including text, figures, and tables. It defines Single Page Single Region, Single Page Multi Regions, and Multi Pages Multi Regions. The human benchmark contains 1,623 QA pairs from 80 papers across eight arXiv categories. A scalable training set contains 30,780 QA pairs from 3,671 papers. Its pipeline renders PDFs at 300 dpi, segments pages with SAM3, filters and groups regions with a VLM, and uses GPT-5.2 or Gemini 3 to generate and verify questions.
The proposed Grounding–Crop–then–Answer protocol first asks a model to predict boxes, crops those regions, and then sends the crops to a fixed Qwen3.5-27B answerer. [email protected], 0.5, and 0.7 measure localization. Most evaluated VLMs remain below 40% IoU, showing that scientific evidence localization is still a major bottleneck; SciEGQA training substantially improves scientific reasoning. The contribution is therefore both a dataset and an evaluation philosophy: measure whether the model found the right evidence before judging its answer. However, the supplied excerpt does not include complete per-model scores, hyperparameters, or ablations.
Deep Analysis
Background
Multimodal language models have advanced image QA and document understanding. DocVQA, MP-DocVQA, and MMLongBench-Doc study page or long-document comprehension; SciDQA, PeerQA, and TAT-DQA target scientific papers; SPIQA, Charxiv, and AnaFig focus on tables, charts, or figures; VisualMRC, DUDE, and BoundingDocs use token-level supervision. Scientific papers combine all these modalities and cross-page links, so no prior granularity simultaneously preserves semantic context and supports precise, multi-page evidence localization.
Core Problem
Given a scientific document D and question Q, a system should produce answer A and supporting regions E. Page labels are too coarse because one page contains many technical elements. Token labels are spatially precise but fragment paragraphs and compositional meaning. Component-only tasks omit broader context. The central bottleneck is therefore selecting semantically complete regions and integrating them across text, tables, figures, and pages before answering.
Innovation
- ��Semantic-region boxes occupy an intermediate granularity between pages and tokens.
- ��The benchmark provides 1,623 human QA pairs from 80 papers, while the training set provides 30,780 automatically constructed pairs from 3,671 papers.
- ��SPSR, SPMR, and MPMR explicitly scale reasoning from one region to cross-page evidence.
- ��Grounding–Crop–then–Answer separates localization from reasoning through a fixed Qwen3.5-27B answerer.
- ��SAM3-based visual segmentation avoids rigid OCR/layout assumptions.
Methodology
- ��Collection: arxiv-dl downloads papers from cs, econ, eess, math, physics, q-bio, q-fin, and stat; PDFs are rendered as 300-dpi PNG pages.
- ��Human annotation: 12 annotators mark coherent text, figure, and table regions. Two annotate independently; IoU below 0.9 triggers third-person review. Twenty-four domain experts design and verify questions.
- ��Segmentation: SAM3 produces visual candidate regions; area thresholds remove extremely small or large segments.
- ��Filtering/grouping: a strong VLM retains semantically meaningful regions and groups them by spatial proximity and semantic relation.
- ��Generation/verification: GPT-5.2 or Gemini 3 generates SPSR, SPMR, and MPMR QA pairs; a VLM re-answers from the supplied evidence, retaining only answerable pairs.
- ��Evaluation: models output boxes in [0,1000] coordinates; [email protected]/0.5/0.7 and answer accuracy are reported. Cropped regions are passed to Qwen3.5-27B.
Experiments
The benchmark has 80 papers, 1,941 pages, and 1,623 QA pairs, with 10 papers from each of eight arXiv categories. The training set has 3,671 papers, 42,380 pages, and 30,780 QA pairs. Two tasks are evaluated: Grounding–Crop–then–Answer and Evidence-Granularity-QA, which compares full-document, evidence-page, and cropped-region inputs. Metrics are answer accuracy and IoU@τ for τ∈{0.3,0.5,0.7}. The excerpt states that many state-of-the-art VLMs are tested but omits the complete suite and ablation settings.
Results
The human set distributes across SPSR, SPMR, and MPMR at 46.15%, 34.26%, and 19.59%; the training set uses 37.91%, 24.41%, and 37.69%, emphasizing more cross-page supervision. Most existing VLMs achieve below 40% IoU, making localization a clear weakness. The fixed answerer controls for generation differences and reveals whether errors arise from wrong boxes or failed reasoning. The paper reports substantial reasoning gains after SciEGQA training, but the supplied text provides no exact per-model improvement values.
Applications
A literature assistant can retrieve a paper, highlight supporting paragraphs, tables, and figures, and generate answers with auditable page coordinates. Research groups and enterprise knowledge bases can use the protocol to detect unsupported summaries. Deployment requires PDF rendering, robust region segmentation, coordinate normalization, multi-page retrieval, and provenance display. Scientific publishers could also use region-linked answers for educational search and technical review workflows.
Limitations & Outlook
The dataset is drawn from open-access arXiv papers and may overrepresent English, computational science, and particular layouts; 80 human-annotated papers are insufficient for broad scientific coverage. SAM3 and frontier VLMs introduce segmentation and generation bias, while automatic verification tests answerability rather than factual validity. Region cropping and multi-page processing add computation. Future work should expand disciplines, verify equations and charts, publish complete baselines and ablations, and train models jointly for grounding, reasoning, and calibrated uncertainty.
Plain Language Accessible to non-experts
Imagine a huge library where every research paper is a room. A conventional assistant tells you only which room contains the answer. That is not enough: the room may contain many shelves, diagrams, tables, and unrelated notes. Another assistant points to a few isolated words, like tearing a sentence into scraps of paper; the scraps lose the explanation that makes them meaningful.
SciEGQA asks the assistant to behave like a careful librarian. It must draw a box around a complete useful item—a paragraph, a table, or a figure. If the question needs two items on the same page, or clues from different pages, it must mark all of them. Then it is allowed to answer only from the marked material.
SAM3 helps find possible visual areas, while GPT-5.2 and Gemini 3 create and check practice questions. During testing, the model first draws boxes; the system cuts out those parts and gives them to Qwen3.5-27B to answer. This separates “I found the wrong shelf” from “I read the right shelf incorrectly.” Most current assistants score below 40% overlap when marking evidence, but training with SciEGQA improves scientific reasoning. The next challenge is covering more subjects, formulas, and complicated diagrams without trusting automatically generated questions too much.
ELI14 Explained like you're 14
Think of a school research-paper game. You see a page packed with tiny writing, charts, and tables. The teacher asks, “What does this paper prove?” You cannot just shout an answer—you must highlight the exact places that prove it. Sometimes one paragraph is enough; sometimes you need a chart plus a paragraph, or clues from pages 2 and 5.
That is SciEGQA. It is a question set that stores both the answer and the highlighted evidence. The highlights are meaningful chunks, not random individual words. The researchers made 1,623 carefully checked questions from 80 papers, then built 30,780 extra practice questions from 3,671 papers using computer helpers.
One helper, SAM3, searches for useful shapes on the page. GPT-5.2 and Gemini 3 write questions and check whether the highlighted parts really contain the answer. In the test, a model first draws boxes; the system cuts out the boxes and asks Qwen3.5-27B for the final answer.
Here is the surprising part: many powerful models can sound smart but still highlight the wrong evidence. Their box overlap is usually below 40%. Training on this dataset makes their science reasoning better. It is like improving not only your quiz score, but also your ability to show the exact line in the textbook that proves your answer!
Glossary
Semantic Region Grounding
A method that marks a complete meaningful document unit with a bounding box. Technically, it provides spatial evidence supervision without fragmenting the unit into isolated tokens.
This is SciEGQA’s central annotation design.
SPSR / SPMR / MPMR
Single Page Single Region, Single Page Multi Regions, and Multi Pages Multi Regions. They represent increasing requirements for evidence composition.
They define the benchmark’s reasoning scenarios.
SAM3
A visual segmentation model that proposes regions directly from page appearance. It reduces dependence on fixed OCR or document-layout rules.
It generates candidates for the automatic training pipeline.
Intersection over Union (IoU)
The intersection area divided by the union area of predicted and reference boxes. Higher IoU means more accurate spatial overlap.
SciEGQA reports [email protected], [email protected], and [email protected].
Grounding–Crop–then–Answer
A protocol that first predicts evidence boxes, crops them, and then answers from the crops. It disentangles evidence localization from answer reasoning.
It is the paper’s principal evaluation protocol.
Evidence-Granularity-QA
An evaluation comparing answers from the full document, evidence pages, and cropped evidence regions. It measures how input scope affects performance.
It is the second benchmark task.
Open Questions Unanswered questions from this research
- 1 The excerpt does not report complete VLM-by-VLM answer accuracy, IoU values, or gains across SPSR, SPMR, and MPMR. Thus the relative strengths of models and the exact benefit of training remain unresolved.
- 2 Automatically generated questions may inherit subject, layout, or generator bias. Expert audits and factual verification are needed to determine whether evidence-supported answers are scientifically correct, not merely answerable.
Applications
Immediate Applications
Auditable scientific QA
Researchers can request an answer together with page coordinates and highlighted paragraphs, tables, or figures. A deployment needs PDF rendering, region detection, provenance display, and uncertainty warnings; the expected benefit is faster manual verification and fewer unsupported summaries.
Literature-reading assistant
A system can retrieve relevant pages, identify multiple evidence regions, and answer comparison questions over papers. Labs and technical teams can integrate it into knowledge bases, provided original citations remain visible and generated claims are checked.
Long-term Vision
Reliable scientific agents
Future agents could jointly retrieve evidence, interpret formulas and charts, reason across pages, and produce fully traceable reports. Obstacles include factual validation, long-context cost, domain transfer, segmentation errors, and bias in automatically generated supervision.
Abstract
Scientific documents contain complex multimodal structures, which makes evidence localization and scientific reasoning in Document Visual Question Answering particularly challenging. However, most existing benchmarks evaluate models only at the page level without explicitly annotating the evidence regions that support the answer, which limits both interpretability and the reliability of evaluation. To address this limitation, we introduce SciEGQA, a scientific document question answering and reasoning dataset with semantic evidence grounding, where supporting evidence is represented as semantically coherent document regions annotated with bounding boxes. SciEGQA consists of two components: a **human-annotated fine-grained benchmark** containing 1,623 high-quality question--answer pairs, and a **large-scale automatically constructed training set** with over 30K QA pairs generated through an automated data construction pipeline. Extensive experiments on a wide range of Vision-Language Models (VLMs) show that existing models still struggle with evidence localization and evidence-based question answering in scientific documents. Training on the proposed dataset significantly improves the scientific reasoning capabilities of VLMs. The project page is available at https://yuwenhan07.github.io/SciEGQA-project/.