MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA

The evolution of multimodal long document QA has seen significant progress, starting from early OCR-based models like Xu et al. (2020, 2021), which fused textual, layout, and visual signals. These models faced challenges with long inputs due to limited context windows. Recent retrieval-augmented methods (Guo et al., 2025; Yu et al., 2024) improved scalability by fetching relevant chunks, but struggled with cross-page and cross-modal reasoning. Agent-based frameworks (Sun et al., 2025; Han et al., 2025) introduced planning and tool invocation, enhancing exploration but still relying on monolithic context accumulation, which leads to information dilution. Memory mechanisms (Wu et al., 2025b; Xu et al., 2025) began addressing long-term reasoning, but lacked explicit structuring tailored for long, multimodal documents. The current challenge is to balance information retention, noise reduction, and reasoning coherence across multiple interaction rounds.

Existing systems often accumulate all interaction traces into a single, ever-growing context, causing key evidence to become scattered and diluted. This hampers multi-hop reasoning, especially over long, multimodal documents with sparse evidence distribution. The core challenge is to develop mechanisms that retain essential facts and logical dependencies while filtering out irrelevant noise. Achieving this balance is crucial for improving accuracy, robustness, and interpretability in long-document QA. Without effective memory management, models tend to hallucinate facts or lose critical information, limiting their practical utility in real-world scenarios like legal analysis, scientific review, and enterprise document management.

The main innovations include:

• �� A structured memory system that separately stores evidence facts and reasoning chains, updated iteratively to maintain relevance.
• �� An explore-refine-reflect cycle that decouples retrieval, evidence compression, and evaluation, reducing noise accumulation.
• �� Multi-modal retrieval tools that integrate visual and textual cues, improving evidence relevance.
• �� A reflection mechanism that assesses evidence sufficiency and guides subsequent retrievals, ensuring logical coherence.

These innovations collectively enable the model to perform more accurate multi-hop reasoning over long, complex documents, addressing the limitations of prior monolithic context approaches.

• �� Construct a hierarchical outline of the document to serve as a retrieval index.
• �� In each iteration, the Explorer uses multi-modal tools to retrieve relevant evidence based on the current query, outline, and memory.
• �� The interaction traces from Explorer are compressed by the Refiner into two structured components: Evidence Memory (facts) and Reasoning Memory (chains).
• �� The memory is updated each cycle, replacing the full interaction history, thus reducing noise.
• �� The Reflector evaluates whether the evidence suffices for answer generation; if not, it provides targeted feedback.
• �� The process repeats for a fixed number of iterations or until the Reflector confirms sufficiency.
• �� Final answer is generated based on the refined, structured memory, ensuring factual accuracy and logical consistency.

The experiments utilized datasets MMLongBench-Doc and DocBench, comprising long, multimodal documents with diverse layouts. Baselines included end-to-end models, retrieval-augmented models, and agent frameworks. Metrics involved accuracy and F1 scores, evaluated by GPT-4o. Hyperparameters included iteration counts (up to 3), backbone models (Qwen3-VL-30B and 8B), and structured memory modules. Ablation studies tested the impact of removing the Refiner and Reflector, confirming their importance. Results showed that MARDoc achieved 57.1% accuracy, outperforming baselines by 8-10%. The experiments validated the structured memory's role in maintaining facts and reasoning chains, especially as evidence pages increased.

MARDoc with Qwen3-VL-30B reached 57.1% accuracy, surpassing traditional models like DocAgent (49.1%). Its performance remained stable as evidence pages increased, demonstrating robustness. Ablation studies revealed that removing the Refiner or Reflector caused over 10% performance drops, highlighting their critical roles. The structured memory effectively filtered noise, preserved facts, and maintained logical chains, enabling multi-hop reasoning over long, multimodal documents. These results confirm that decoupling retrieval and evidence compression enhances reasoning quality and model robustness.

This framework can be applied to financial report analysis, scientific literature review, legal document understanding, and enterprise knowledge management. Its ability to handle long, multimodal data with high factual fidelity makes it suitable for critical decision-making tasks in industry. Future integration with external knowledge bases and reinforcement learning could further extend its capabilities, enabling autonomous, scalable AI systems for complex document understanding across various domains.

Despite promising results, the approach faces challenges with extremely large or highly complex documents, where evidence may be insufficient or noisy. Computational overhead from memory maintenance limits real-time deployment. The current design also relies on predefined iteration limits, which may not adapt well to all scenarios. Future work should focus on optimizing memory compression, dynamic iteration strategies, and broader multimodal fusion to enhance scalability and robustness in diverse real-world applications.