KoViDoRe: Korean Visual Document Retrieval

Multimodal retrieval has rapidly advanced, with models like ColPali, ViDoRe, and Jina-VDR leveraging deep learning to combine visual, textual, and layout cues for complex document understanding. These approaches have significantly improved retrieval accuracy for PDFs, reports, and forms, primarily in English and European languages. However, Korean documents pose unique challenges due to script complexity, morphological variation, and diverse layouts. Existing datasets mostly focus on single-page retrieval or simplified formats, limiting their applicability to real-world scenarios where information spans multiple pages and involves complex visual structures. As enterprise and government sectors demand more sophisticated document understanding, there is a pressing need for benchmarks and models tailored to Korean multi-page, multimodal documents. This gap hampers progress in multilingual AI and restricts deployment in Korean contexts, underscoring the importance of specialized datasets and evaluation frameworks.

Current models struggle with multi-page, structured Korean documents, especially when relevant information is dispersed across pages. Existing benchmarks focus on single-page retrieval, failing to evaluate models' ability to aggregate evidence and perform multi-hop reasoning. This limits their practical utility in real-world applications like legal, financial, and administrative document processing. The core challenge is designing a benchmark that accurately reflects the complexity of Korean documents, including multi-page layouts, visual cues, and diverse query types requiring cross-page inference. Without such benchmarks, models cannot be effectively trained or evaluated for real-world deployment, creating a bottleneck in advancing multilingual multimodal retrieval systems.

This work introduces several innovations: 1) a multi-stage synthetic query generation pipeline combining summary- and context-based strategies to produce diverse, complex queries aligned with multi-page evidence needs; 2) a structural parsing method (Upstage Document Parse) tailored for Korean documents, extracting detailed layout information including text blocks, tables, and figures; 3) a relevance annotation framework integrating automatic and manual verification to ensure high-quality multi-page relevance labels; 4) a large-scale Korean-specific training dataset (Ko-VDR Train Public) that enhances model adaptation to complex layouts and visual content. These innovations collectively enable more realistic evaluation and training of models for Korean multi-page document retrieval, addressing critical gaps in existing research.

• �� Data collection: Gather diverse Korean documents from public sources, including government reports and enterprise materials.
• �� Structural parsing: Use Upstage Document Parse to extract fine-grained layout features such as text blocks, tables, figures, and captions.
• �� Query generation: Employ SolarPro to generate complex, reasoning-oriented queries based on document summaries and local contexts, emphasizing multi-hop, numerical, and cross-page inference.
• �� Relevance mapping: Combine automatic relevance scoring with human verification to label query-document page pairs, filtering low-quality or trivial cases.
• �� Dataset construction: Generate a large set of query-page pairs, applying rule-based filters and manual review to ensure high relevance accuracy.
• �� Model evaluation: Test multiple models (e.g., ColPali, CLIP variants, Qwen3) using nDCG@10, analyzing performance across domains and model sizes.

The evaluation involves testing various multimodal models on the KoViDoRe benchmark, focusing on their ability to retrieve relevant pages in multi-page, structured documents. The models include small (e.g., CLIP-base), medium (e.g., Qwen3-VL-Embedding-2B), and large (e.g., Qwen3-VL-Embedding-8B) architectures. Experiments compare baseline performance with models fine-tuned on the large-scale Korean dataset (Ko-VDR Train Public), measuring improvements in nDCG@10. Additional ablation studies analyze the impact of query generation strategies, structural parsing accuracy, and training data size. The evaluation covers four domains—Cybersecurity, Energy, Economic, and HR—highlighting domain-specific challenges and model robustness.

The top-performing model (Qwen3-VL-Embedding-8B) achieves 50.4% nDCG@10, outperforming smaller models by a significant margin. Fine-tuning on the large-scale Korean dataset boosts performance by 20-30%, especially in multi-page, structured scenarios. Models show notable difficulty in economic and HR domains, where information is highly dispersed. Results confirm that multi-page evidence aggregation remains a key bottleneck, and larger models alone cannot fully address the challenge. The experiments validate the effectiveness of the synthetic query pipeline and structural parsing in improving retrieval accuracy.

This research enables enterprise and governmental organizations to develop intelligent document management systems capable of multi-page, multimodal content understanding. Applications include automated report analysis, legal document retrieval, and policy document summarization, reducing manual effort and increasing efficiency. The framework can be integrated into existing knowledge management platforms, supporting multilingual environments and complex layouts. Long-term, it paves the way for fully automated, context-aware document assistants that can handle diverse, real-world data with high accuracy.

Despite advances, models still struggle with highly complex, dispersed multi-page documents, especially when visual cues are ambiguous or layouts are unconventional. Synthetic queries, while diverse, do not fully replicate real user needs, limiting ecological validity. Computational costs for training large models remain high, posing deployment challenges. The current architecture lacks explicit mechanisms for cross-page reasoning, which could be addressed by integrating graph neural networks. Future work should focus on improving reasoning robustness, reducing resource consumption, and incorporating real user interaction data for more natural query understanding.