MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects
MUDDLE evaluates document understanding by separating distractor and length effects, improving gpt-5-mini accuracy.
Key Findings
Methodology
MUDDLE uses 270 human-annotated questions, each linked to a single document, tested under five conditions: source alone, two or four topically similar hard negatives, and two or four random distractors. LLM evaluation analyzes model performance under these conditions.
Key Results
- In markdown, gpt-5-mini accuracy drops by 0.030 to 0.041 with hard negatives compared to random distractors.
- Random distractors' accuracy stays near the no-distractor baseline, showing minimal length impact.
- For gpt-5-mini, hard negatives significantly underperform length-matched random distractors.
Significance
MUDDLE provides a controlled benchmark for document QA systems, separating distractor and length effects, aiding researchers in understanding model performance in complex document environments.
Technical Contribution
MUDDLE introduces a multi-document evaluation suite, varying distractor type and count while keeping the source document fixed and matching context length across distractor types.
Novelty
MUDDLE is the first to separate topical similarity from length effects, offering a refined evaluation framework to identify weaknesses in document QA systems.
Limitations
- Currently, full five-condition evaluation is only in markdown; image and PDF conditions are not yet tested at scale.
- Length matching has some error, potentially affecting result accuracy.
Future Work
Future work will extend to image and PDF conditions, increase distractor count, and introduce genuinely unrelated distractors for a more conservative baseline.
AI Executive Summary
MUDDLE addresses the common confusion between distractor and length effects in document QA systems. Existing systems often fail to distinguish whether errors are due to excessive context or overly similar distractors. MUDDLE creates a controlled benchmark to separate these effects. This benchmark uses 270 questions, each linked to a single document, tested under five conditions: source alone, two or four topically similar hard negatives, and two or four random distractors. Experimental results show that in gpt-5-mini's markdown condition, hard negatives significantly lower accuracy compared to random distractors, indicating that topical similarity has a greater impact on accuracy. MUDDLE's contribution lies in providing a refined evaluation framework for document QA systems, helping researchers identify weaknesses in complex document environments. Future work will extend to image and PDF conditions and increase distractor count to further validate results.
Deep Analysis
Background
With the development of large language models, document QA systems have shifted from reading single documents to handling collections of retrieved documents. However, these systems often perform poorly when faced with lengthy or distracting information. Existing research often conflates context length and distractor effects, making it difficult to accurately assess system robustness.
Core Problem
Document QA systems often fail when processing multiple documents due to excessive context length or overly similar distractors. Separating these two effects to accurately assess system performance is a pressing issue.
Innovation
MUDDLE separates distractor topical similarity from length effects, providing a refined evaluation framework. Its innovation lies in analyzing the impact of different types of distractors on system performance while keeping context length consistent.
Methodology
- �� Uses 270 human-annotated questions, each linked to a single document.
- �� Tests under five conditions: source alone, two or four topically similar hard negatives, and two or four random distractors.
- �� Evaluates model performance using LLM, analyzing different conditions.
Experiments
Experiments conducted in markdown condition using gpt-5-mini, gemini-3.5-flash, and grok-4.3 models. Evaluation metrics include semantic correctness, exact match, and F1 score.
Results
Results show that hard negatives have a greater impact on gpt-5-mini accuracy, especially under k=4 conditions, where accuracy significantly drops. Random distractors have minimal impact, staying near the no-distractor baseline.
Applications
MUDDLE can be used to evaluate document QA systems' performance in complex document environments, helping researchers identify model weaknesses and guide system improvements.
Limitations & Outlook
Current evaluation is only conducted in markdown; image and PDF conditions are not yet tested at scale. Additionally, length matching has some error, potentially affecting result accuracy.
Plain Language Accessible to non-experts
Imagine you're in a library searching for a specific book, but there are many similar-looking books on the shelves. MUDDLE acts like a smart librarian, helping you filter out those that look similar but are unrelated. It analyzes the books' topics and lengths to help you find your target faster. It's like finding the person you're talking to in a noisy room. MUDDLE helps systems understand documents more accurately by separating distractor and length effects.
ELI14 Explained like you're 14
Imagine you're playing a mystery game where you need to find clues from a pile of seemingly related information. MUDDLE is like a super helper, removing misleading fake clues. It analyzes each piece of information's topic and length to ensure you focus only on the important parts. Just like finding the right exit in a maze, MUDDLE helps systems find answers faster and more accurately.
Glossary
Distractor
In document QA, a distractor is a document related to the question but not containing the answer.
MUDDLE analyzes distractor topic and length to evaluate their impact on system performance.
Hard Negative
Documents similar in topic but not containing the answer.
MUDDLE uses hard negatives to test model robustness in complex document environments.
markdown
A lightweight markup language for formatting text.
MUDDLE conducts full five-condition evaluation in markdown.
Semantic Correctness
Evaluates the accuracy of model answers in terms of semantics.
MUDDLE uses LLM to evaluate model semantic correctness under different conditions.
Context Length
In document QA, it refers to the total length of text processed by the model.
MUDDLE matches context length to analyze the impact of different types of distractors.
Open Questions Unanswered questions from this research
- 1 How to conduct large-scale evaluations in image and PDF conditions?
- 2 How to automate the manual review process for hard negatives?
Applications
Immediate Applications
Document QA System Evaluation
Researchers can use MUDDLE to evaluate system performance in complex document environments, identifying model weaknesses.
Long-term Vision
Intelligent Information Retrieval
Improve information retrieval accuracy and efficiency by enhancing document QA systems.
Abstract
Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation.