LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
LiveBrowseComp shows that search agents often verify memory: on 335 fresh-fact questions, closed-book accuracy stayed below 2%.
Key Findings
Methodology
The paper uses three diagnostics on BrowseComp, BrowseComp-ZH, HLE, and GAIA: closed-book answering, evidence-blocked search, and trajectory grounding. Evidence blocking is implemented with BrowseComp-Plus, where Qwen3-8B-Embedding builds a dense index after gold and supporting documents are removed. A query is model-originated when its key information first appears in the model’s reasoning rather than retrieved results.
Key Results
- Across 24 model–benchmark pairs, closed-book pass@4 ranges from 20.4 to 62.0, averaging 38.9. MiniMax M2.5 reaches 44.5 on BrowseComp and Kimi-K2.6 reaches 62.0 on BrowseComp-ZH without tools.
- Blocking answer-supporting evidence reduces average pass@4 from 26.1 to 6.2, below the closed-book baseline for every model. MiniMax M2.5 falls from 44.5 to 8.0, while Kimi-K2.6 falls from 25.5 to 2.3.
- More than half of queries are model-originated, exceeding 60% later in browsing. After useful evidence is retrieved, agents use it in only 24.7%–32.2% of cases, indicating predominantly hypothesis-led search.
Significance
The work separates two capabilities usually conflated by static search benchmarks: what a model already knows and what it can discover from the web. As training and post-training absorb old benchmark facts, scores increasingly reward memory-backed verification. LiveBrowseComp provides a more realistic test of information discovery, with implications for agent research, model selection, and high-stakes deployment.
Technical Contribution
The contribution is a controlled diagnostic framework rather than a new search algorithm. It decomposes tool value into intrinsic coverage, evidence necessity, and query provenance. BrowseComp-Plus supplies gold, evidence, irrelevant, and hard-negative documents for intervention. LiveBrowseComp uses timestamped records from GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS, followed by temporal, long-tail, stability, and multi-stage human verification filters.
Novelty
The paper turns the question of whether agents genuinely search into measurable diagnostics and systematically shows that search can underperform closed-book answering when support is removed. Its benchmark combines recent facts, low salience, multi-hop reasoning, and archived snapshots, addressing a weakness of static BrowseComp-style evaluation.
Limitations
- The 335-question dataset is limited to six structured sources and English questions, so it may not represent the full diversity of web search.
- Query provenance depends on trajectory annotations and identifying when key information first appears; incomplete reasoning traces can introduce ambiguity.
- The offline dense index and unified scaffold reduce variance but cannot fully reproduce live-engine ranking, crawling failures, page changes, or tool diversity.
Future Work
Future work should expand languages, domains, and continuously versioned sources while preserving reproducibility. Stronger causal trajectory analysis could test when agents update beliefs from evidence. Promising directions include evidence verification, active exploration, recovery after failed searches, calibrated uncertainty, citation faithfulness, and metrics that report intrinsic coverage separately from search gain.
AI Executive Summary
LLM search agents are expected to browse the web, combine evidence, and solve complex information needs. Yet a high score on a static benchmark may show only that a model remembers the answer. LiveBrowseComp investigates this possibility through three tests: answering without tools, searching after supporting documents are removed, and tracing whether later queries originate in retrieved evidence or internal hypotheses.
The resulting failure mode is called Intrinsic Knowledge Dependence (IKD). Across 24 model–benchmark pairs, closed-book pass@4 averages 38.9 and reaches 62.0 in one case. MiniMax M2.5 scores 44.5 on BrowseComp with no retrieval. When supporting evidence is blocked, average performance collapses from 26.1 to 6.2; MiniMax M2.5 drops to 8.0 and Kimi-K2.6 to 2.3. More than half of queries originate from model reasoning, while retrieved supporting evidence is used only 24.7%–32.2% of the time.
To evaluate search beyond memory coverage, the authors introduce LiveBrowseComp, a 335-question benchmark built from facts published within the preceding 90 days and filtered for low global salience. Seeds come from GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS. Every evaluated model achieves below 2% closed-book accuracy; search scores fall roughly 25–40 points relative to BrowseComp. Human solve rates are nearly identical, 30% versus 31%, suggesting that the model drop reflects loss of the memory shortcut rather than excessive task difficulty. The benchmark reframes search evaluation as discovery, not confirmation.
Deep Analysis
Background
Search evaluation has progressed from TriviaQA and NaturalQuestions to HotpotQA, BrowseComp, and DeepSearchQA. Systems such as OpenAI Deep Research and Gemini Deep Research demonstrate practical promise. However, facts in static datasets can enter model parameters through training and updates. Final scores therefore combine prior knowledge with retrieval ability, obscuring whether an agent actually depends on web evidence.
Core Problem
The central question is whether tool use creates new evidence or merely validates a pre-existing hypothesis. A robust agent should preserve a correct answer or change strategy when retrieval fails. Instead, the studied agents are often diverted by hard negatives and continue searching around internally generated entities. Standard accuracy cannot distinguish these behaviors.
Innovation
The paper contributes two innovations. First, IKD and its three diagnostics make hidden dependence measurable: closed-book coverage, evidence blocking, and query provenance. Second, LiveBrowseComp contains recent, obscure, multi-hop questions whose answers cannot be recovered from older information. Human evidence chains and independent checks enforce correctness, uniqueness, difficulty, and genuine temporal dependence.
Methodology
- �� Closed-book: disable every tool and measure pass@4 on BrowseComp, BrowseComp-ZH, HLE, and GAIA.
- �� Evidence blocking: use Qwen3-8B-Embedding to index BrowseComp-Plus, removing gold and supporting documents while retaining irrelevant and hard-negative items.
- �� Trajectory grounding: identify whether query information first appears in model reasoning or retrieved documents, then check evidence use within the next three rounds.
- �� Benchmark construction: collect timestamped events from GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS; apply 90-day, long-tail, and stability filters; construct and independently verify unique answers.
Experiments
Models include GLM-5.0/5.1, MiniMax M2.5, Kimi-K2.5/2.6, and DeepSeek-V4-Pro. A shared search-agent scaffold controls interaction protocol, sampling budget, context limit, and answer format. Metrics include pass@4, tool gain, blocked-search degradation, model-originated query rate, and evidence-use rate. Separate human trials compare BrowseComp and LiveBrowseComp solve rates and completion times.
Results
Closed-book pass@4 spans 20.4–62.0, averaging 38.9. Evidence blocking reduces average performance from 26.1 to 6.2. MiniMax M2.5 falls from 44.5 to 8.0 and Kimi-K2.6 from 25.5 to 2.3. Model-originated queries exceed 50% and later exceed 60%; evidence use remains 24.7%–32.2%. On LiveBrowseComp, all models score below 2% closed-book and lose roughly 25–40 points versus BrowseComp, while human solve rates remain 30% and 31%.
Applications
Developers can use the diagnostics to determine whether an agent fails at retrieval, evidence integration, or resistance to misleading documents. Organizations in news monitoring, cybersecurity, market intelligence, and fact-checking can use recent long-tail questions to test discovery of novel facts, source triangulation, and uncertainty calibration before deployment.
Limitations & Outlook
The dataset is small, English-only, and tied to six structured sources. A 90-day window lowers but cannot eliminate exposure through model updates. The offline index is not a live search engine. Future evaluations should add languages, domains, dynamic snapshots, and real web tools, while developing agents that detect uninformative results, pivot after failed searches, and maintain evidence-grounded uncertainty.
Plain Language Accessible to non-experts
Imagine a student asked to solve a difficult investigation. In the first exam, the answer appeared in an old textbook. The student guesses from memory, then searches the internet for a sentence that agrees. The result looks like research, but the web did not actually produce the answer. In a second exam, the teacher removes the useful pages and leaves only similar but misleading material. A careful student should admit uncertainty or change direction; many tested agents instead follow the misleading clues.
LiveBrowseComp is like an investigation about events that happened very recently: a new software vulnerability, a little-known film record, a sports event, or an earthquake. The student must find fresh clues and connect several pages. Humans solve these questions at about the same rate as comparable older questions, but models lose much of their advantage.
The lesson is simple: searching is not typing more keywords. It means learning something genuinely new from evidence, checking whether the evidence supports the answer, and abandoning the first guess when it does not.
ELI14 Explained like you're 14
Think of an AI as a detective in a game. You ask, “Which year did this software vulnerability happen?” If the vulnerability appeared last month, memorizing old facts will not help. A strong detective searches databases, news, and technical reports, then connects the clues.
Many AIs do something different: they guess first and search second. On old benchmark questions, this trick works because the facts may already be inside the model. Without search, MiniMax M2.5 still gets 44.5 on BrowseComp, and Kimi-K2.6 gets 62.0 on the Chinese version.
Researchers then kept the search button but removed the pages that really supported the answers. Average performance fell from 26.1 to 6.2. Some models became worse than when they were not searching! More than half of their queries came from their own guesses, and they often ignored useful evidence.
So the researchers built LiveBrowseComp with 335 questions about fresh facts from the last 90 days. Every model scored below 2% without tools. The real challenge is not remembering yesterday’s answer; it is finding something new, checking the proof, and changing your plan when your first idea is wrong!
Glossary
Intrinsic Knowledge Dependence (IKD)
A search agent relies on knowledge encoded in its parameters to generate answers or hypotheses before retrieval. Search then functions mainly as confirmation rather than discovery.
The paper uses IKD to explain high static-benchmark scores and collapse under evidence blocking.
Closed-book pass@4
The probability that at least one of four sampled answers is correct when all tools are disabled. It is a conservative proxy for intrinsic knowledge coverage.
It estimates how much benchmark performance exists before browsing begins.
Evidence-blocked search
The search interface remains available, but documents supporting the correct answer are removed from retrieval. Only irrelevant and hard-negative documents remain.
This intervention tests whether search helps without confirmation evidence.
Model-originated query
A query whose key information first appears in the model’s own reasoning rather than in retrieved results. It measures whether search is hypothesis-led.
The trajectory study finds that such queries exceed half of all queries.
LiveBrowseComp
A 335-question deep-search benchmark based on recent, low-salience facts and multi-step evidence chains. It is designed to suppress parametric-memory shortcuts.
It is the paper’s proposed alternative to static BrowseComp evaluation.
Open Questions Unanswered questions from this research
- 1 How can evaluators prove that an agent updated its belief from evidence rather than merely reinterpreting new pages to fit its initial guess? Better causal interventions and transparent state tracking are needed.
- 2 Can a 90-day freshness rule resist continual model updates over time? Dynamic, archived, multilingual benchmarks are required to test this sustainably.
Applications
Immediate Applications
Search-agent auditing
Model teams can report closed-book coverage, blocked-search degradation, query provenance, and evidence use alongside accuracy. This reveals whether an agent genuinely retrieves information or mainly confirms internal guesses.
Real-time fact and vulnerability research
Newsrooms, security teams, and intelligence analysts can use recent long-tail questions to test whether agents find new facts, triangulate sources, and express uncertainty instead of forcing familiar answers onto novel events.
Long-term Vision
Dynamic evaluation standards
Future benchmarks could continuously generate timestamped questions while preserving web snapshots. Rankings would then measure evidence discovery, strategic pivoting, and citation reliability rather than mostly reflecting training-data coverage.
Abstract
Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.