MARCA: A Checklist-Based Benchmark for Multilingual Web Search

Recent advances in LLMs like GPT-4, PaLM, and Claude have demonstrated impressive capabilities in language understanding, but their web search and evidence synthesis abilities still lag behind human performance, especially in multilingual contexts. Early benchmarks such as WebGPT and BrowseComp focused on English, evaluating models' ability to navigate web pages and retrieve relevant info. However, these benchmarks lacked coverage of non-English languages, particularly Portuguese, which is spoken by over 250 million people. The challenge lies in models' ability to understand diverse web content, handle cross-lingual queries, and accurately integrate multi-source evidence. Existing datasets like mMARCO and MIRACL evaluate passage retrieval but do not assess end-to-end web interaction or multi-entity reasoning. As multilingual AI applications expand, there is a pressing need for standardized, comprehensive benchmarks that evaluate models' real-world web search performance across languages and complex multi-entity questions.

Despite progress, current evaluation methods inadequately measure models' ability to perform multi-entity, multi-web source reasoning in multilingual settings. Static datasets cannot capture dynamic web content or real-time information updates. Moreover, models often struggle with cross-lingual retrieval, entity disambiguation, and evidence verification, leading to incomplete or incorrect answers. The lack of a unified, fine-grained evaluation framework hampers targeted improvements and fair comparison across models. Addressing these gaps requires a benchmark that simulates realistic web search scenarios, incorporates multi-language content, and provides detailed, granular performance metrics, enabling researchers to diagnose specific weaknesses and develop more robust multilingual retrieval systems.

MARCA introduces several innovations: 1) a bilingual, multi-entity question set with manual annotations covering 9 domains, 2) a checklist-based evaluation rubric that explicitly measures answer completeness and correctness, 3) two interaction frameworks—basic and orchestrator—that simulate different web search strategies, 4) multiple execution runs per question to quantify stochasticity, and 5) leveraging GPT-4 as a judge for fine-grained scoring. This comprehensive approach allows detailed performance analysis across languages, models, and interaction paradigms, providing insights into the strengths and weaknesses of current systems in complex web-based reasoning tasks. The combination of manual question design, dynamic checklists, and multi-framework evaluation sets MARCA apart from prior static or monolingual benchmarks.

• �� Develop 52 multi-entity questions across 9 domains, each paired with a manually curated checklist specifying expected entities and attributes.
• �� Implement two evaluation frameworks: (i) a basic setup where models directly invoke web_search and web_scrape tools, and (ii) an orchestrator setup where a top-level model decomposes questions into sub-queries delegated to subagents.
• �� Execute each question three times, capturing variability, and use GPT-4 to judge whether answers satisfy checklist criteria.
• �� Calculate checklist accuracy as the ratio of satisfied checklist items to total, aggregating results across runs with mean and standard deviation.
• �� Compare performance across languages (English and Portuguese) and frameworks, analyzing the impact of task decomposition and multilingual transfer.
• �� Conduct cost-performance analysis based on token usage and accuracy to inform practical deployment considerations.

The experiments used a custom dataset of 52 questions, covering topics like awards, sports, and history, evaluated across 14 models including GPT-4, Gemini-3, and Qwen-3. Each model was tested under both frameworks and in both languages, with three runs per question. The primary metric was checklist accuracy, with detailed scoring by GPT-4. Additional analyses examined the effect of task decomposition, language transferability, and cost-performance trade-offs. The results highlighted that models like GPT-4 and Gemini-3-pro achieved near 90% accuracy, while weaker models hovered around 26%. The orchestrator framework generally improved performance, especially for models with difficulty managing multi-entity retrieval. Cross-lingual performance varied, with some models showing significant drops in Portuguese, indicating challenges in multilingual web search. Cost analysis revealed a trade-off between accuracy and computational expense, guiding practical deployment decisions.

Model performance varied widely, with top models reaching ~0.90 checklist accuracy, while weaker ones scored below 0.30. The orchestrator framework consistently improved accuracy by 0.05-0.20 points, especially for models struggling with multi-entity tasks. Multilingual transferability was inconsistent; some models like GPT-4 maintained performance across languages, whereas others like Qwen-3-235B experienced over 5% drops in Portuguese. Cost analysis showed higher accuracy models incur significantly higher token usage, emphasizing the trade-off between performance and resource consumption. These findings underscore the importance of task decomposition and language-specific tuning in real-world applications.

MARCA can be used to benchmark and improve multilingual web search systems, especially for applications requiring multi-entity reasoning, such as intelligent virtual assistants, multilingual search engines, and content curation tools. It helps developers identify weaknesses in cross-lingual retrieval and evidence aggregation, guiding targeted model fine-tuning. In industry, it supports the deployment of more reliable, fair, and accurate multilingual AI systems, enhancing user trust and satisfaction. Additionally, the benchmark can inform academic research on multi-source reasoning, multi-lingual information retrieval, and AI explainability, fostering innovations in complex web-based AI reasoning.

The reliance on GPT-4 as the judgment tool may introduce bias and limit objectivity. The benchmark's scope is limited to 52 questions, which may not cover all real-world scenarios. The cost and computational demands of the orchestrator framework pose challenges for large-scale deployment. The dynamic web content and language nuances may still cause inaccuracies, especially in low-resource languages. Future work should address scalability, diversify question sets, and incorporate real-time web content to better reflect practical conditions.