Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents

Institutional documents serve as vital sources of operational, analytical, and policy information across humanitarian, governmental, and development sectors. Historically, research in document layout analysis focused on academic papers and standardized formats, with datasets like PubLayNet and DocLayNet facilitating progress through supervised learning. Recent advances leverage transformer-based models such as LayoutLM and multimodal architectures like TF-ID-Large, which combine textual, visual, and spatial cues to improve understanding. Despite these developments, real-world institutional documents pose unique challenges: layouts are highly heterogeneous, visual elements are densely packed, and semantic content often depends on contextual cues. These factors limit the effectiveness of existing models, which are primarily optimized for academic datasets, leading to a gap between research and practical deployment. Addressing this gap requires specialized benchmarks and models tailored to operational needs, emphasizing semantic relevance and spatial accuracy in complex layouts.

The core challenge lies in accurately detecting and localizing analysis-rich visual regions—referred to as data snapshots—in diverse institutional documents. Traditional layout detection models excel at geometric localization but lack the semantic discernment necessary to distinguish meaningful analytical artifacts from irrelevant visual clutter such as logos, decorative images, or formatting elements. This results in high false positive rates, fragmented detections, and missed critical contextual cues like titles, legends, or footnotes. The problem is compounded by the variability in document structures, visual density, and language, making it difficult for generic models to generalize effectively. Consequently, automated systems struggle to reliably extract operational insights, impeding downstream tasks like data integration, knowledge graph construction, and decision support. Developing models that can balance spatial precision with semantic understanding remains a pressing and complex problem.

This work introduces several key innovations:

• �� A multi-source, high-quality dataset encompassing humanitarian, policy, and project documents, annotated specifically for analysis-rich visual artifacts.
• �� A novel evaluation framework that combines detection metrics (Precision, Recall, IoU) with spatial quality measures (Area Recall, Area Precision), emphasizing semantic completeness.
• �� The formal definition of 'data snapshots' as semantically meaningful visual regions containing structured or semi-structured information, guiding model focus beyond mere geometric detection.
• �� Benchmarking of four open-source models—TF-ID-Large, DocLayout-YOLO, YOLOv11, and YOLOv26—highlighting their respective strengths and weaknesses in operational scenarios.
• �� Insights into the failure modes of current models, such as fragmentation, confusion with irrelevant content, and incomplete contextual extraction, informing future research directions.

• �� Data collection: Curated a diverse corpus of nearly 8,000 pages from humanitarian reports, policy papers, and project documents, ensuring coverage of various layouts and visual styles.
• �� Annotation process: Employed semi-automatic pre-labeling using existing models, followed by meticulous manual review and correction via Label Studio, ensuring high annotation fidelity.
• �� Definition of data snapshots: Focused on visual regions containing analytical content like tables, charts, and geospatial maps, including contextual elements when necessary.
• �� Model selection: Chose transformer-based (TF-ID-Large) and YOLO-based models (YOLOv11, YOLOv26) to cover different detection paradigms.
• �� Training: Pre-trained models on academic datasets, then fine-tuned on the institutional dataset to adapt to domain-specific layouts.
• �� Evaluation: Employed standard object detection metrics (Precision, Recall, IoU) and proposed spatial metrics (Area Recall, Area Precision) to assess both detection accuracy and semantic completeness.
• �� Post-processing: Applied area filtering to remove small irrelevant detections, improving practical detection quality.

• �� Dataset split: Divided data into training, validation, and testing subsets, ensuring diverse representation.
• �� Hyperparameter tuning: Adjusted learning rates, batch sizes, and augmentation strategies for each model to optimize performance.
• �� Evaluation metrics: Calculated Precision, Recall, IoU, Area Recall, and Area Precision across all models and datasets.
• �� Comparative analysis: Assessed models’ ability to detect and accurately localize analysis regions, identifying common failure modes such as fragmentation and misclassification.
• �� Ablation studies: Tested the impact of different post-processing filters and training strategies on detection performance.
• �� Cross-scenario testing: Evaluated models on documents with varying complexity, layout density, and visual styles to gauge robustness.

• �� TF-ID-Large achieved the highest spatial IoU (0.877) and Area Recall (93.8%), indicating excellent spatial coverage but lower detection precision (0.628), suggesting some false positives.
• �� YOLOv11 and YOLOv26 models demonstrated higher recall (up to 0.893) but lower IoU (~0.817-0.824), reflecting more boundary inaccuracies and fragmentation issues.
• �� All models struggled with complex, densely packed layouts, often fragmenting composite analysis regions or missing contextual cues like titles and legends.
• �� Post-processing filtering improved precision but revealed persistent challenges in balancing sensitivity and specificity, especially in cluttered documents.
• �� The results highlight the need for models that incorporate semantic understanding to improve the quality of extracted analysis regions, moving beyond purely geometric detection.

• �� Automated extraction of key visual insights from government, NGO, and corporate reports to accelerate decision-making and policy formulation.
• �� Rapid retrieval and summarization of analytical content in academic and technical documents, supporting research workflows.
• �� Integration into enterprise document management systems for structured data harvesting, reducing manual effort.
• �� Future integration with multimodal models could enable end-to-end semantic understanding, facilitating intelligent document analysis platforms.
• �� Broader impact includes enabling smarter knowledge bases, automated compliance checks, and real-time monitoring systems across sectors.

• �� Current models exhibit fragmentation and misclassification in complex, cluttered layouts, limiting their immediate deployment in operational environments.
• �� Evaluation primarily focuses on spatial overlap metrics, lacking comprehensive semantic interpretability assessments.
• �� Dataset scope, while diverse, remains limited in size and scope, necessitating expansion to include more document types, languages, and visual styles.
• �� Computational costs for training and inference remain high, especially for transformer-based models, posing challenges for large-scale deployment.
• �� Future work must address robustness, interpretability, and scalability to fully realize practical applications.