PosterSum: A Multimodal Benchmark for Scientific Poster Summarization

The evolution of scientific communication increasingly relies on visual and textual integration, exemplified by scientific posters that combine dense information with visual aids. Prior works like LLaVA and MiniCPM have advanced multimodal understanding but mainly target general tasks like image captioning or visual question answering. These models lack specialization for dense, structured scientific content, especially in complex layouts. The absence of dedicated datasets and methods for scientific poster summarization has limited progress. As conferences grow larger, the need for automated tools to interpret and summarize posters becomes urgent, prompting research into domain-specific multimodal models that can handle intricate layouts and technical content efficiently.

The core challenge lies in the complexity of scientific posters, which feature densely packed text, multiple visual elements, and intricate layouts. Existing models often fail to accurately interpret these multimodal, structured documents, leading to incomplete or inaccurate summaries. The main bottleneck is the inability to effectively segment, understand, and synthesize diverse content types within a unified framework. This hampers the deployment of automated summarization tools in real-world academic settings, where precision and interpretability are critical. Addressing these issues requires innovative architectures that can decompose the problem into manageable sub-tasks, capturing the detailed semantics of each region before integrating them into a comprehensive abstract.

The paper introduces a hierarchical segmentation-based approach, which is novel in scientific poster summarization. Key innovations include: 1) a segmentation model that divides posters into meaningful regions aligned with their structure; 2) a multimodal understanding module that extracts textual and visual content from each region; 3) a local summarization step that captures fine-grained details; 4) a global synthesis step that merges regional summaries into a coherent abstract. This layered architecture addresses the limitations of end-to-end models by explicitly modeling the poster’s structure, improving interpretability and accuracy. The approach leverages pre-trained models, avoiding additional training, and demonstrates robustness across diverse poster layouts.

• �� Use a pre-trained segmentation model Sϕ to divide the poster into multiple regions based on visual cues.
• �� Cluster similar regions with a clustering algorithm C to reduce redundancy and ensure coverage.
• �� For each region, apply a multimodal large model Vϕ to extract content—text, images, charts—and generate a localized summary.
• �� Collect all regional summaries and input them into a text-based large language model Lω.
• �� The language model synthesizes these into a final, cohesive abstract, maintaining logical coherence.
• �� The entire pipeline operates without additional fine-tuning, relying on pre-trained knowledge.
• �� The process emphasizes structural cues, enabling detailed understanding of dense, multimodal content.

The evaluation uses the PosterSum dataset, comprising 16,305 poster-abstract pairs from top AI conferences. Metrics include ROUGE, BERTScore, SacreBLEU, and METEOR, providing a comprehensive assessment of accuracy and fluency. Baselines include OCR-based extraction, end-to-end multimodal models, and fine-tuned approaches. Hyperparameters are standardized across models, with beam search decoding and 10 epochs for fine-tuning. Ablation studies test the impact of clustering and regional understanding. Results show the hierarchical method outperforms all baselines, with ROUGE-L scores exceeding 24%, demonstrating robustness across diverse poster types and content complexities.

The hierarchical segmentation and summarization approach achieves a ROUGE-L score of 24.18, surpassing the best baseline by 3.14%. It also outperforms models like GPT-4o and LLaVA-NeXT across all metrics, including SacreBLEU and BERTScore. Ablation results confirm clustering improves performance by reducing redundancy, while regional understanding enhances detail capture. Fine-tuning with LoRA further boosts scores, indicating adaptability. The approach effectively handles complex layouts, dense text, and multimodal content, validating its potential for real-world scientific applications.

This technology can be integrated into conference platforms, enabling automatic poster summarization for reviewers and attendees. It facilitates rapid comprehension of research, aiding in peer review and dissemination. In industry, it supports automated report generation, knowledge extraction, and digital library indexing. The method’s scalability and interpretability make it suitable for diverse scientific domains, including biomedical, engineering, and social sciences, where complex multimodal documents are prevalent.

The approach depends on the quality of pre-trained segmentation and understanding models; poor image quality or unconventional layouts can impair performance. Computational costs are non-trivial, limiting real-time deployment. The method currently focuses on academic posters; adaptation to other document types requires further research. Additionally, the absence of fine-tuning may limit performance in highly specialized domains, necessitating future domain-specific training or adaptation.