QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Query-based multi-domain meeting summarization with locate-then-summarize framework; QMSum dataset highlights challenges in long meeting understanding.
Key Findings
Methodology
The study introduces a two-stage approach: first, a locator (pointer network and hierarchical ranking model) identifies relevant segments; second, pre-trained models like BART and HMNet generate summaries. The QMSum dataset includes 232 meetings with 1808 query-summary pairs, annotated hierarchically for multi-granularity evaluation. Experiments show the locator achieves ROUGE-L scores up to 84.04%, outperforming baselines, while summarization models reach ROUGE-1 of 29.20%. Challenges remain in factual consistency and detail retention.
Key Results
- The locator model achieved ROUGE-L recall of 84.04%, significantly surpassing random and similarity baselines. Summarization models like BART attained ROUGE-1 scores of 29.20% and ROUGE-2 of 6.37%, indicating effective content compression but still facing factuality issues. Multi-domain models generalize well across fields but struggle with complex queries and diverse topics. Human evaluation highlights persistent factual errors and incomplete details, emphasizing the need for improved reasoning and verification.
Significance
This work advances personalized, multi-granularity meeting understanding, providing a large-scale dataset and robust models for diverse user needs. It addresses the core challenge of summarizing lengthy, multi-topic discussions, fostering progress in long-text comprehension and generation. The multi-domain aspect enhances model robustness, supporting real-world applications such as intelligent meeting assistants and enterprise knowledge management, ultimately transforming how organizations process and utilize meeting data.
Technical Contribution
The core innovation lies in the locate-then-summarize framework, combining pointer networks and hierarchical ranking for effective span localization, integrated with powerful pre-trained models for abstractive summarization. The hierarchical annotation structure enriches training signals, enabling models to handle multi-granularity content. The approach balances efficiency and performance, setting a foundation for future multi-modal, multi-task meeting understanding systems.
Novelty
This is the first large-scale dataset covering multiple domains with hierarchical annotations for query-based meeting summarization. The proposed locate-then-summarize architecture uniquely combines span localization with abstractive generation, addressing the limitations of previous single-structure models. The multi-granularity and multi-domain design significantly extend the scope of automated meeting understanding research.
Limitations
- Models still face factual inaccuracies and incomplete details, especially in multi-topic, complex queries. High computational costs hinder real-time deployment. Data annotation relies on manual efforts, risking bias and inconsistency. Future work should focus on improving factual verification, efficiency, and expanding multimodal integration.
Future Work
Future directions include integrating audio and video modalities for richer context, developing more efficient and accurate models, and scaling datasets for multilingual and multi-scenario applications. Enhancing factual consistency and reasoning capabilities remains a priority, aiming to realize fully autonomous, personalized meeting assistants.
AI Executive Summary
Meetings are fundamental to human collaboration, yet their increasing volume and complexity pose significant challenges for effective information retrieval. Traditional summarization methods often produce generic overviews, failing to meet diverse user needs. To address this, the paper introduces a novel query-based multi-domain meeting summarization task, supported by the newly constructed QMSum dataset, comprising 232 meetings and 1808 query-summary pairs with hierarchical annotations.
The proposed framework employs a locate-then-summarize approach. First, a locator—using pointer networks and hierarchical ranking models—identifies relevant segments within lengthy transcripts. Then, powerful pre-trained models like BART and HMNet generate concise, query-specific summaries. Experimental results demonstrate that the locator achieves ROUGE-L scores up to 84.04%, significantly outperforming baselines. Summarization models attain ROUGE-1 scores of 29.20%, indicating promising content compression but revealing persistent issues with factual correctness and detail preservation.
This research marks a step forward in personalized, multi-granularity meeting understanding. The multi-domain dataset facilitates evaluating model robustness across diverse scenarios, fostering advancements in long-text comprehension and generation. The framework’s modular design allows future integration of multimodal data, such as audio and video, to further enhance understanding. Despite progress, challenges remain in ensuring factual accuracy and computational efficiency, guiding future research directions. Overall, this work lays a solid foundation for intelligent, adaptable meeting summarization systems, with broad implications for enterprise knowledge management and AI-assisted collaboration.
Deep Analysis
Background
The evolution of meeting summarization reflects the growing need for efficient information extraction from lengthy, multi-topic discussions. Early datasets like AMI and ICSI provided foundational resources but lacked multi-domain coverage and hierarchical annotations. Deep learning models, especially Transformer-based architectures like BART, have significantly advanced the field, enabling abstractive summarization. However, these models struggle with long transcripts, multi-topic content, and personalized query responses. Recent efforts focus on multi-modal data and hierarchical structures, yet comprehensive datasets supporting multi-domain, multi-granularity evaluation remain scarce. This gap limits progress in personalized, scalable meeting understanding systems.
Core Problem
Existing summarization approaches predominantly generate global summaries, insufficient for users seeking specific information. Long meeting transcripts contain diverse topics, decisions, opinions, and actions, making it difficult for models to locate relevant content and produce concise, accurate summaries. The challenge intensifies across multiple domains, where models must generalize beyond training data. Moreover, current models often produce factually inconsistent outputs and lack interpretability. Addressing these issues requires new datasets, architectures, and evaluation metrics tailored to personalized, query-based summarization of lengthy, multi-topic meetings.
Innovation
The paper introduces several key innovations: 1) a large, multi-domain dataset (QMSum) with hierarchical annotations supporting multi-granularity queries; 2) a locate-then-summarize framework combining pointer networks and hierarchical ranking models for effective span localization; 3) integration of state-of-the-art pre-trained models (BART, HMNet) for abstractive summarization, tailored for long transcripts. These innovations enable models to better understand and extract relevant segments, improving personalization and scalability. The hierarchical annotation structure enhances interpretability and training efficiency, setting a new standard for multi-domain, query-focused meeting summarization.
Methodology
- �� Data collection: Meetings from academic, product, and committee domains, annotated hierarchically with main topics and relevant spans. • Query design: General and specific queries crafted to reflect diverse user interests, with constraints on span length. • Span localization: Using pointer networks and hierarchical ranking models to identify relevant segments based on queries. • Summarization: Applying models like BART and HMNet to generate abstractive summaries from localized spans. • Training: Multi-task learning with hierarchical supervision, optimizing span accuracy and summary quality. • Evaluation: ROUGE metrics and human assessments to measure informativeness, factuality, and relevance.
Experiments
The dataset was split into training, validation, and test sets, ensuring domain diversity. Baselines included random, similarity-based, pointer network, and hierarchical ranking models for localization, combined with PGNet, BART, and HMNet for summarization. Hyperparameters were tuned for each model, with ablation studies conducted to assess component contributions. Experiments evaluated span localization accuracy via ROUGE-L, and summarization quality through ROUGE scores and human judgments. Results confirmed the effectiveness of the locate-then-summarize approach, with detailed analysis of strengths and limitations across scenarios.
Results
The locator achieved ROUGE-L scores up to 84.04%, significantly surpassing baselines. Summarization models like BART and HMNet achieved ROUGE-1 scores of 29.20% and 32.29%, respectively, demonstrating strong content compression. Human evaluation revealed issues with factual consistency and detail retention, especially in complex queries. Cross-domain tests indicated good generalization, but challenges in maintaining factuality persisted. Ablation studies highlighted the importance of hierarchical annotations and multi-task training, guiding future improvements.
Applications
This framework can be deployed in enterprise meeting systems, enabling personalized, query-based summaries for diverse stakeholders. It supports rapid information retrieval, decision tracking, and knowledge management. Future integration with multimodal data (audio, video) will enhance contextual understanding, broadening applications in virtual conferencing, legal proceedings, and remote collaboration. The approach also paves the way for AI-powered assistants capable of real-time, multi-faceted meeting comprehension.
Limitations & Outlook
Models still face factual inaccuracies, especially with complex, multi-topic content. High computational costs hinder real-time deployment. Manual annotation introduces bias and limits scalability. The approach requires further refinement to handle multimodal inputs and improve reasoning. Future work must address these issues to realize fully autonomous, reliable meeting understanding systems.
Plain Language Accessible to non-experts
想象你在一个大型厨房里准备一顿盛宴。每个厨师负责不同的菜肴，厨房里有很多食材和步骤。有时候你只关心某一道菜的做法，比如甜点或者素食。于是，你会先找到相关的食材（定位），然后用简短的话告诉你怎么做（摘要）。这个过程就像会议中，有很多不同的话题和内容。系统会帮你找到你感兴趣的部分（定位），再用简洁的话语告诉你重点（摘要），让你不用看一堆繁杂的内容，就能快速了解会议的重点。这就像厨房里找到所有需要的食材，然后快速做出你想吃的菜。
ELI14 Explained like you're 14
想象你在学校的食堂点餐。有很多不同的菜，有的你喜欢，有的你不感兴趣。现在，你只想知道“今天的甜点”或者“素食菜单”。那么，食堂的工作人员会帮你找到这些信息（定位），然后告诉你具体内容（摘要）。这就像会议里，大家关心不同的话题。系统会先帮你找到你关心的部分，然后用简短的话告诉你重点。这让你不用看一大堆繁杂的内容，就能快速知道自己想知道的事情。
Abstract
Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at \url{https://github.com/Yale-LILY/QMSum}.