Query-Focused Event Summarization: A Dataset and Benchmark
Proposed QFES task and QFESum dataset; combined RAT and SHC frameworks outperform baselines in event-focused summarization.
Key Findings
Methodology
This paper introduces a two-stage QFES framework: first, Query-Focused Retrieval with Adaptive Thresholding (RAT) employs BGE and LLM verification to filter relevant documents; second, Query-Focused Summarization based on Hierarchical Clustering (SHC) extracts events, clusters them, and resolves coreferences. The QFESum dataset, built from T17 and CRISIS, contains 8 thematic events, 16,684 documents, and 104 queries, annotated with relevance and event information. The framework leverages LLMs' semantic understanding and hierarchical clustering techniques to handle large, complex corpora, improving event coverage and relevance in summaries.
Key Results
- On QFESum, the combined RAT and SHC approach achieved ROUGE-1/2/L scores of 38.71/10.44/16.08, BLEU and METEOR scores showed significant improvements over baselines, and LLM-F1 reached 12.21, indicating strong event relevance and content conciseness.
- Compared to GraphRAG and FG-RAG, the proposed method demonstrated superior event matching accuracy and reduced redundancy, especially in large-scale datasets, with better coverage and fewer irrelevant details.
- Human evaluation confirmed that the method's event coverage exceeded 75%, with higher relevance and fewer extraneous details, validating its practical effectiveness.
Significance
This work addresses the critical gap in event-oriented query summarization datasets and methods, offering a scalable, effective framework that combines retrieval and event clustering. It advances the state-of-the-art in personalized, event-focused information extraction, with broad implications for news analysis, crisis management, and policy decision support. The dataset and framework lay a foundation for future research in multi-modal, multi-source event understanding, fostering more intelligent and efficient information systems.
Technical Contribution
Key innovations include the adaptive retrieval threshold mechanism, which dynamically adjusts relevance criteria using LLM verification, and the hierarchical clustering-based event selection and coreference resolution, which enhance event coherence and relevance. The integration of these components with large language models enables robust handling of large-scale, complex data, providing a new paradigm for event-focused summarization that balances precision, coverage, and computational efficiency.
Novelty
This is the first comprehensive system to construct an event-centric query-focused summarization dataset and develop a two-stage framework combining adaptive retrieval and hierarchical event clustering. Unlike prior work that either emphasizes generic multi-document summarization or shallow query relevance, this approach explicitly models event semantics and relevance, setting a new benchmark for large-scale, event-oriented summarization.
Limitations
- The framework's performance declines in highly ambiguous or multi-faceted events due to limitations in event extraction and clustering granularity. Computational costs are high, especially with multiple LLM verifications, limiting real-time deployment.
- Current models lack explicit temporal and spatial reasoning, which are crucial for complex event understanding. Incorporating multi-modal data (images, videos) remains a future challenge.
- Handling evolving events with rapid updates or multi-language data requires further adaptation of the current system.
Future Work
Future directions include integrating multi-modal data to enrich event representations, optimizing retrieval and clustering efficiency for real-time applications, and extending the dataset to cover more diverse domains and languages. Additionally, enhancing temporal and spatial reasoning capabilities and improving model interpretability will be key to broader adoption and deployment.
AI Executive Summary
In an era overwhelmed by information, efficiently extracting relevant event content from massive data sources is crucial. Traditional multi-document event summarization methods focus on content condensation but often neglect user-specific queries and relevance. To address this, we propose the Query-Focused Event Summarization (QFES) task, aiming to generate concise, query-relevant event summaries from large thematic corpora. We constructed the QFESum dataset, derived from T17 and CRISIS, encompassing 8 thematic events, 16,684 documents, and 104 queries, with detailed annotations for relevance and event content. This dataset provides a rich resource for developing and evaluating event-focused summarization models.
The core of our approach is a two-stage framework combining Query-Focused Retrieval with Adaptive Thresholding (RAT) and Hierarchical Clustering-based Summarization (SHC). RAT employs dense retrieval via BGE and a dynamic relevance threshold, refined through LLM verification, to efficiently filter relevant documents. SHC then extracts events, clusters them hierarchically, and performs coreference resolution to generate a coherent, focused event summary. This design effectively handles large-scale, complex data, ensuring high relevance and coverage.
Experimental results on QFESum demonstrate that our method surpasses strong baselines, achieving ROUGE scores of 38.71/10.44/16.08 and an LLM-F1 of 12.21, with human evaluations confirming superior event coverage and relevance. The approach reduces redundancy, improves event coherence, and adapts well to diverse thematic scenarios. These advances significantly contribute to personalized, event-centric information extraction, with promising applications in news analysis, crisis response, and policy decision-making.
Despite its strengths, the framework faces limitations such as high computational costs and challenges in modeling complex temporal or multi-modal events. Future work aims to incorporate multi-modal data, optimize efficiency for real-time deployment, and expand coverage across domains and languages. Overall, this research marks a substantial step toward intelligent, scalable event understanding and summarization, with broad implications for both academia and industry.
Deep Dive
Abstract
A thematic corpus is a collection of semantically coherent documents that collectively describe different aspects of a shared thematic event. Such a corpus typically contains hundreds or even thousands of documents. While users' interests in a thematic event often span multiple dimensions, Query-Focused Summarization (QFS) aims to generate summaries tailored to users' queries. However, existing QFS datasets lack event-oriented summarization, and most QFS methods struggle with large-scale corpora. To address these challenges, we propose the Query-Focused Event Summarization (QFES) task and construct the QFESum dataset, which contains 8 thematic events, 16,684 documents, and 104 queries. Furthermore, we introduce a two-stage QFES framework consisting of Query-Focused Retrieval with Adaptive Thresholding (RAT) and Query-Focused Summarization based on Hierarchical Clustering (SHC). Experimental results on QFESum show that RAT and SHC consistently outperform the baselines, demonstrating their effectiveness for QFES. The dataset and code are publicly available at https://github.com/sarcasm-hcy02/QFES-QFESum.