UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs
UniSumEval evaluates 9 LLMs on fine-grained, multi-dimensional summarization across 9 domains, addressing gaps in existing benchmarks.
Key Findings
Methodology
UniSumEval introduces a benchmark covering 9 domains (e.g., news, dialogue, reports) with AI-assisted fine-grained annotations for faithfulness, completeness, and conciseness. It ensures high inter-annotator agreement (IAA).
Key Results
- Result 1: GPT-4-turbo achieved the highest faithfulness score (97.3%) and domain stability across all models.
- Result 2: Dialogue-based inputs outperformed non-dialogue ones, especially in completeness for long texts.
- Result 3: PII redaction caused an average 2.1% drop in faithfulness across all models.
Significance
UniSumEval fills a critical gap by providing a multi-dimensional, fine-grained benchmark for summarization evaluation, enabling more robust assessments for academia and industry.
Technical Contribution
Key contributions include: 1) AI-assisted annotation for improved consistency on long texts; 2) new evaluation dimensions like domain stability and PII redaction impact; 3) systematic comparison of SOTA evaluators across diverse scenarios.
Novelty
UniSumEval is the first benchmark to support fine-grained, multi-dimensional evaluation, particularly for faithfulness, completeness, and conciseness.
Limitations
- Limitation 1: AI-assisted annotations may introduce bias, especially for complex texts.
- Limitation 2: Dataset size is limited, potentially missing broader domain diversity.
- Limitation 3: PII redaction experiments are restricted to specific domains.
Future Work
Future work could expand domain coverage, refine AI-assisted annotation workflows, and explore more efficient automated evaluators.
AI Executive Summary
Evaluating summarization quality is a persistent challenge in NLP. Existing benchmarks often focus on single domains or coarse-grained annotations, limiting their applicability. UniSumEval addresses these gaps by introducing a fine-grained, multi-dimensional benchmark covering 9 domains, including news, dialogue, and reports.
The study employs AI-assisted annotation to improve consistency for long texts, achieving high inter-annotator agreement (Krippendorff’s α = 0.60). Evaluating 9 state-of-the-art LLMs, including GPT-4-turbo and Claude2.1, the results show GPT-4-turbo excels in faithfulness (97.3%) and domain stability, while dialogue-based inputs perform better in completeness. PII redaction negatively impacts faithfulness, with an average 2.1% drop.
UniSumEval provides a robust tool for developing and evaluating advanced summarization models. Future research could expand its domain coverage and optimize AI-assisted workflows, paving the way for more precise automated evaluations.
Deep Analysis
Background
Text summarization has advanced significantly with LLMs, but existing benchmarks like SummEval and FRANK are limited to single domains (e.g., news) or coarse-grained annotations, failing to capture diverse real-world scenarios.
Core Problem
Current benchmarks lack input diversity, fine-grained annotations, and comprehensive evaluation dimensions, making it difficult to assess model performance in complex scenarios like long texts or dialogues.
Innovation
UniSumEval introduces: 1) coverage of 9 domains with diverse text types and lengths; 2) AI-assisted annotation for improved consistency; 3) new evaluation dimensions like domain stability and PII redaction impact.
Methodology
- �� Data sourcing: Sampled 200 texts from 9 datasets (e.g., CNN/DM, MultiWOZ).
- �� Summary generation: Used 9 models, including GPT-4-turbo and Claude2.1.
- �� Annotation: AI-assisted labeling for faithfulness, completeness, and conciseness.
- �� Evaluation: Measured sentence-level faithfulness, key-fact completeness, and conciseness.
Experiments
The study evaluated 2,025 text-summary pairs across 9 domains. Human annotations assessed faithfulness, completeness, and conciseness, comparing model performance under diverse scenarios.
Results
GPT-4-turbo excelled in faithfulness (97.3%) and domain stability. Dialogue-based inputs showed higher completeness scores, while PII redaction caused a 2.1% drop in faithfulness.
Applications
UniSumEval can guide the development of precise automated evaluators and support industrial applications like customer service and legal document summarization.
Limitations & Outlook
Limitations include potential annotation bias, limited dataset size, and restricted applicability of PII redaction experiments.
Plain Language Accessible to non-experts
Imagine you're in a library trying to summarize a book. Existing tools are like casual reviewers—they might miss key points or add errors. UniSumEval acts like a professional team: one checks facts, another ensures all key points are included, and a third removes unnecessary details. The result? A summary that's accurate, complete, and concise.
ELI14 Explained like you're 14
Think of writing a book report for school. Regular tools are like your classmates—they might miss important parts or get things wrong. UniSumEval is like the class genius who checks every fact, includes all key points, and avoids extra fluff. It's like getting an A+ every time!
Glossary
Faithfulness
Measures whether the summary aligns factually with the source text.
Used to evaluate sentence-level factual accuracy.
Completeness
Checks if the summary includes all critical information from the source.
Assessed via key-fact validation.
Conciseness
Evaluates whether the summary avoids redundant information.
Measured through key-fact alignment.
Domain Stability
Consistency of model performance across different domains.
Used for cross-domain performance analysis.
PII Redaction
Hiding personally identifiable information (e.g., phone numbers).
Tested for privacy-sensitive scenarios.
Open Questions Unanswered questions from this research
- 1 How can AI-assisted annotation accuracy be improved?
- 2 How to expand dataset coverage to more domains?
- 3 How to mitigate the impact of PII redaction on performance?
Applications
Immediate Applications
Customer Service Summaries
Generate accurate summaries of customer interactions to improve service efficiency.
Legal Document Processing
Help legal professionals extract key information from contracts or rulings.
Long-term Vision
Universal Text Evaluation Framework
Develop a unified tool for evaluating all text generation tasks.
Abstract
Existing benchmarks for summarization quality evaluation often lack diverse input scenarios, focus on narrowly defined dimensions (e.g., faithfulness), and struggle with subjective and coarse-grained annotation schemes. To address these shortcomings, we create UniSumEval benchmark, which extends the range of input context (e.g., domain, length) and provides fine-grained, multi-dimensional annotations. We use AI assistance in data creation, identifying potentially hallucinogenic input texts, and also helping human annotators reduce the difficulty of fine-grained annotation tasks. With UniSumEval, we benchmark nine latest language models as summarizers, offering insights into their performance across varying input contexts and evaluation dimensions. Furthermore, we conduct a thorough comparison of SOTA automated summary evaluators. Our benchmark data will be available at https://github.com/DISL-Lab/UniSumEval-v1.0.