Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics
A 35B MoE-based document understanding model with difficulty-aware data curation reduces deployment costs by over 80%.
Key Findings
Methodology
This paper introduces a 35B parameter Mixture-of-Experts (MoE) model integrated with a Difficulty-Aware Data Curation (DADC) pipeline. The model employs sparse expert activation, enabling efficient multi-task learning for fields extraction, classification, visual validation, and VQA. Training combines internal production data with filtered open-source PDFs, ensuring data challenge and diversity. The system uses prompting for multi-task adaptation, deployed on a single H100 GPU, meeting strict latency and cost constraints. The DADC pipeline filters samples based on difficulty, improving data quality and model robustness. A quality-adjusted cost framework quantifies performance versus economic efficiency, guiding deployment decisions.
Key Results
- The model achieves an average score of 0.814 across internal benchmarks, outperforming baseline models by 5.3 points with only a fraction of parameters. On public benchmarks like DocVQA and InfoVQA, it surpasses previous models significantly, especially in multilingual transfer. Cost analysis shows over 80% savings compared to human annotation and 50% compared to larger models, with deployment latency below 10 seconds.
- Ablation studies confirm the effectiveness of DADC in filtering challenging samples, enhancing generalization. The model maintains high throughput and cost-efficiency across different workflows, demonstrating industry applicability.
- The approach outperforms larger models in multi-task, multi-language scenarios, validating the importance of data difficulty management combined with sparse model architecture for real-world deployment.
Significance
This work addresses the critical industry challenge of deploying high-performance document understanding models under strict hardware, privacy, and cost constraints. It demonstrates that careful data curation and cost-aware modeling can drastically reduce operational expenses while maintaining accuracy. The methodology provides a scalable, quantifiable framework for evaluating model deployment economics, facilitating industry adoption of large multimodal models. It bridges the gap between academic advances and practical industry needs, enabling automation in regulated sectors like finance and legal services.
Technical Contribution
The paper’s key technical innovations include integrating MoE architecture with a difficulty-aware data filtering pipeline, and developing a cost-performance quantification framework based on production telemetry. These enable scalable training on challenging, representative samples, and precise economic evaluation of deployment feasibility. The unified prompting interface allows multi-task, multi-process deployment on minimal hardware, setting a new standard for industry-ready multimodal models.
Novelty
This is the first work to combine sparse MoE models with difficulty-aware data filtering tailored for privacy-restricted, industry-specific document understanding. Unlike prior models focusing solely on scale or generic datasets, this approach emphasizes data challenge management and cost-effectiveness, providing a practical blueprint for deploying large models in regulated environments.
Limitations
- The model primarily targets direct fields extraction, with limited multi-step reasoning or cross-page context handling. Extending capabilities to complex, multi-page workflows remains future work.
- Cross-lingual transfer is validated mainly on Russian and related languages; performance on other languages needs further exploration.
- Despite significant cost savings, large-scale deployment still incurs substantial hardware costs, and further model compression or efficiency improvements are needed for broader adoption.
Future Work
Future directions include integrating reinforcement learning to enhance multi-step reasoning, expanding multi-lingual capabilities, and automating sample difficulty management. Additionally, exploring multi-modal fusion and reasoning over multi-page documents will further improve performance in complex workflows. Continued optimization of data filtering and model compression will facilitate wider industry deployment.
AI Executive Summary
In today’s digital era, automating document understanding is crucial for industries like finance, legal, and government. Traditional approaches rely heavily on OCR combined with manual annotation, which is costly and difficult to scale. While recent advances in vision-language models (VLMs) like Qwen have shown promise, their deployment in regulated sectors faces hurdles such as high computational costs, strict latency requirements, and privacy restrictions.
This paper presents a breakthrough solution—a 35B parameter Mixture-of-Experts (MoE) model integrated with a novel Difficulty-Aware Data Curation (DADC) pipeline. The model is designed to operate efficiently on a single GPU, meeting industry latency constraints while significantly reducing deployment costs. By combining internal production data with carefully filtered open-source PDFs, the approach ensures high-quality, challenging training samples that improve model robustness and generalization.
The core innovation lies in the DADC pipeline, which filters samples based on difficulty, ensuring the model learns from informative and diverse data. This strategy, coupled with a cost-adjusted performance evaluation framework, enables precise quantification of economic benefits. The results demonstrate over 80% cost savings compared to human annotation and more than 50% savings over larger, less efficient models. The model’s performance on both internal benchmarks and public datasets confirms its robustness and generalization capabilities.
Beyond technical excellence, this work offers a practical blueprint for deploying large multimodal models in privacy-sensitive, resource-constrained environments. It paves the way for scalable, cost-effective automation solutions that can transform industries by reducing manual effort, lowering operational costs, and enhancing accuracy. The authors also outline future work to extend reasoning capabilities, improve multi-lingual support, and further optimize efficiency, promising a broad impact on the future of intelligent document processing.
Deep Analysis
Background
随着大规模视觉-语言模型（VLM）在多模态理解中的快速发展，诸如Qwen系列、Baidu-ERNIE等模型在通用任务中取得显著突破。然而，行业应用面临硬件成本高、隐私限制、数据多样性不足等难题。传统的OCR+专家标注流程虽精确但成本昂贵，难以满足大规模、实时的需求。近年来，MoE架构因其参数稀疏性，成为提升模型效率的关键技术，但在行业场景中的应用仍有限。此前研究多集中在通用模型，缺乏针对受隐私限制行业的定制化解决方案。本研究旨在突破行业应用瓶颈，结合困难感知数据筛选，构建高效、低成本的行业级文档理解模型。
Core Problem
行业中大量文档（如法院判决、发票、税务凭证）需要自动提取结构化信息，但受隐私法规限制，不能依赖云端模型，且现有开源VLM在质量上难以满足生产需求。更大模型虽能提升性能，但部署成本高昂，难以实现规模化应用。此外，传统流水线复杂、维护成本高，且难以满足严格的时延要求。如何在保证高精度的同时，降低部署成本、提升效率，成为行业亟待解决的核心问题。
Innovation
本研究提出结合MoE架构与困难感知筛选（DADC）策略的创新方案。首先，采用稀疏专家模型，有效控制参数规模与计算成本。其次，DADC筛选出难度适中的样本，提升训练样本的质量与多样性，增强模型泛化能力。再次，提出基于生产遥测数据的质量调整成本分析框架，将模型性能与经济效益紧密结合。最后，融合内部行业数据与精选开源PDF，突破数据单一限制，实现行业场景的高效适应。
Methodology
- �� 构建35B参数的MoE模型，结合多任务学习，支持字段抽取、分类、视觉验证和问答。
- �� 采集内部生产数据，结合开源PDF，利用DADC筛选难度样本，确保训练样本的挑战性和多样性。
- �� 通过PDF渲染、文本一致性验证、跨模型验证等多阶段筛选，过滤掉低质量或偏差样本。
- �� 利用提示（prompting）技术，将模型调优为单一接口，支持多任务、多流程部署。
- �� 设计质量调整的成本分析模型，将每个字段的准确率转化为经济指标，指导模型选择与调优。
- �� 在单GPU环境下，满足行业严格的时延和成本限制，实现高效部署。
Experiments
采用内部司法和发票数据集进行训练和评估，结合公开基准（如MWS Vision Benchmark、OCRBench等）测试模型的通用性。对比不同模型规模（如Qwen3.5-35B、122B、397B）和训练策略（有无SFT、不同数据源比例）。通过 ablation 研究验证DADC筛选效果，分析不同数据融合比例对性能的影响。指标包括字段准确率、F1、文档分类准确率及成本效益比。模型在单页、多页和细粒度任务中均表现优异，验证了方法的有效性。
Results
模型在内部评估中平均得分0.814，超越基础模型5.3点，参数规模仅为最优模型的1/10，部署成本低于行业标准。公开基准测试中，模型在DocVQA、InfoVQA等任务中提升显著，尤其在多语言迁移方面表现优异。成本分析显示，模型比人类标注节省80%以上，且在满足时延要求下实现高吞吐。ablation结果表明，DADC筛选策略显著提升样本难度和模型泛化能力，验证了其在行业场景中的实用价值。
Applications
该模型适用于金融、司法、税务等行业的自动化信息提取，支持批量处理、实时响应和多任务集成。只需少量硬件资源，即可实现高精度、低成本的部署，帮助企业降低人力成本、提升效率。未来还可扩展到多语种、多模态场景，推动行业数字化转型。
Limitations & Outlook
模型在多步推理、跨页信息整合方面仍有限，未来需结合强化学习和多模态融合技术提升能力。此外，模型对极端低资源或超复杂布局下表现仍有限，未来需持续优化样本难度管理和模型推理能力。硬件成本虽已降低，但大规模部署仍面临一定挑战，未来需探索更高效的模型压缩与优化策略。
Plain Language Accessible to non-experts
想象你在一家大型工厂工作，工厂里有很多不同的机器，每台机器负责不同的任务，比如装配、包装、检验。以前，每个任务都需要专门的机器和工人，成本高、效率低。现在，工厂引入了一台智能机器人，它可以根据任务的难度自动调整工作方式，既能快速完成简单任务，也能应对复杂的检验。这个机器人像模型一样，结合了很多不同的技能（参数），还能根据任务的难易程度选择不同的工作策略。通过优化工厂的流程，既节省了成本，又保证了质量。这个故事反映了论文中提出的智能模型如何通过筛选难度样本、优化成本，帮助行业自动化，降低开支，同时保持高质量。
ELI14 Explained like you're 14
想象你在学校里，有一台超级聪明的老师机器人，它可以帮你做很多事情，比如批改作业、回答问题。以前，老师要花很多时间检查每个学生的作业，成本很高。而这台机器人可以自己判断哪些题目比较难，优先解决难题，简单的题目它就快速处理。它还会根据不同的任务调整自己的学习策略，比如在考试前多练难题。这样一来，老师就可以节省很多时间，学生也能更快得到帮助。论文里的模型就像这个机器人，它通过筛选难度样本，优化学习和工作流程，既节省成本，又保证了高质量的结果。
Glossary
Mixture-of-Experts（MoE）模型（专家混合模型）
一种稀疏激活的神经网络架构，通过多个专家子模型协作完成任务，参数利用率高，计算效率优。
论文中用以构建高效大规模多任务模型。
困难感知数据筛选（DADC）
一种筛选训练样本的策略，根据样本难度筛除易错或低信息量样本，提升模型泛化能力。
用于训练数据的质量控制和样本难度管理。
提示（prompting）
利用预定义的输入提示引导模型完成多任务，减少模型微调需求。
实现模型多任务适应的关键技术。
质量调整成本分析（Quality-Adjusted Cost Analysis）
将模型性能指标转化为经济成本指标，评估模型部署的经济效益。
指导模型选择与优化。
开源PDF数据（Open-source PDFs）
从公共数据源（如Common Crawl）采集的PDF文档，用于模型训练和筛选。
丰富训练样本，提高模型多样性。
Open Questions Unanswered questions from this research
- 1 模型在多步推理和跨页信息整合方面的能力仍有限，未来需结合强化学习和多模态融合技术提升。
- 2 跨语言迁移能力尚未充分验证，尤其在非俄语语系的表现有待探索。
- 3 如何在极端复杂布局和低质量输入环境中保持高性能，是未来研究的重要方向。
Applications
Immediate Applications
行业自动化信息提取
金融、司法等行业可用该模型实现批量文档结构化，降低人工成本，提升效率。
Long-term Vision
行业智能化升级
未来模型将支持多语种、多模态、多任务场景，推动行业全面数字化转型，降低门槛。
Abstract
Extracting structured fields from hundreds of millions of documents annually remains costly in regulated industries: bespoke OCR cascades cover only a fraction of workflows, privacy rules preclude external models, and existing open-source VLMs that clear quality thresholds cost more to serve than human annotation. We present a deployed document-understanding system built on a Mixture-of-Experts VLM (35B total, 3B active), fine-tuned on in-house production data mixed with open-domain documents curated by a Difficulty-Aware pipeline for layout diversity, fact-extractability, and cross-model consistency. Fitting on a single H100 and serving heterogeneous workflows via prompting, the model leads all deployable (non-reasoning) baselines up to an order of magnitude larger. A quality-adjusted cost analysis, with confirmation and correction costs calibrated from production telemetry, shows it reduces expected costs by over 80% against the human baseline and by more than 50% against the best competing open-source model, while larger baselines remain economically unviable.