What's under the hood: Investigating Automatic Metrics on Meeting Summarization

The evolution of meeting summarization has transitioned from keyword extraction to abstractive methods powered by Transformer architectures like BART, PEGASUS, and recent large language models. Early metrics such as ROUGE provided basic content overlap measures but failed to capture the nuanced challenges of meetings, including speaker role dynamics, discourse structure, and implicit context. Recent advances introduced semantic-aware metrics like BERTScore and QA-based methods like QuestEval, yet their effectiveness in meeting scenarios remains underexplored. The complexity of meeting transcripts, with features like long dialogues, turn-taking, and contextual dependencies, demands more sophisticated evaluation strategies. Despite progress, current metrics often overlook critical errors such as hallucinations, structural disorganization, and implicit information loss, leading to unreliable assessments. This gap motivates the present study, aiming to systematically evaluate the sensitivity and limitations of existing metrics in meeting summarization.

The core issue is that existing automatic metrics are insufficiently sensitive to meeting-specific challenges, leading to inaccurate quality assessments. Traditional metrics like ROUGE mainly measure lexical overlap, missing errors like hallucinations or incoherence. More advanced metrics, though promising, often fail to respond adequately to the unique complexities of meetings, such as turn-taking and implicit context. This mismatch hampers model development, as poor evaluation can mislead optimization efforts and deployment decisions. Furthermore, the lack of understanding of how different models (encoder-decoder vs. LLMs) respond to meeting challenges complicates the design of effective evaluation tools. Addressing these gaps is crucial for advancing the field toward more reliable, context-aware assessment frameworks.

The study introduces a novel framework linking human-annotated challenges and errors with automatic metric responses, providing a detailed analysis of their correlations across multiple models. It systematically evaluates nine metrics, revealing their strengths and weaknesses in detecting specific error types. The approach incorporates model architecture differences, highlighting how encoder-decoder and LLMs vary in their error sensitivities. This comprehensive analysis offers new insights into the limitations of current metrics and suggests strategies for multi-metric fusion, aiming to improve overall evaluation robustness. The integration of human annotations with quantitative correlation analysis represents a significant methodological advance in the field.

• �� Literature review to identify key challenges (speaker dynamics, discourse structure, implicit context) and error types (omission, hallucination, incoherence). • Selection of models including LED, DialogLED, PEGASUS-X, GPT-3.5, Zephyr-7B-α, trained on QMSum. • Human annotators identify challenges and errors in 35 samples, ensuring high inter-annotator agreement (Krippendorff's alpha ~0.81). • Automatic metrics (ROUGE, BERTScore, QuestEval, etc.) are computed on generated summaries. • Correlation analysis (point-biserial) assesses the relationship between metrics and human annotations, with further ablation studies on metric combinations.

The experiments utilize the QMSum dataset, covering diverse meeting types (academic, corporate, parliamentary). Models are fine-tuned on training subsets, then generate summaries evaluated against human annotations. The analysis includes correlation studies between nine automatic metrics and annotated errors, focusing on how well each metric detects specific challenges. Ablation experiments test the effectiveness of combining multiple metrics. The evaluation emphasizes the sensitivity of metrics to errors like omissions, hallucinations, and structural disorganization, providing a comprehensive picture of their performance across different model architectures.

• �� Encoder-decoder models show high correlation (0.74-0.87) between challenges like incoherence and errors such as structural disorganization, while LLMs exhibit weaker correlations (<0.3) with hallucinations and omissions.
• �� ROUGE primarily detects omissions but poorly captures coherence issues, with correlations around -0.4, and sometimes masks errors like hallucinations.
• �� Combining metrics such as ROUGE, BERTScore, and LENS improves error detection but no single metric covers all error types effectively, indicating the need for multi-metric evaluation frameworks.

The findings support the development of more accurate evaluation tools for meeting summarization, aiding researchers and industry practitioners in model selection and optimization. Enhanced metrics can improve the reliability of automated meeting minutes, decision summaries, and remote collaboration tools, ultimately leading to better organizational communication and knowledge management.

The study relies on the QMSum dataset, which may not represent all meeting types or languages. Static correlation analysis does not account for dynamic error severity. Only a limited set of models was examined, and future work should explore multimodal data and broader architectures to validate and extend these findings.