MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
MoverScore combines contextual embeddings and Earth Mover's Distance for high semantic correlation evaluation.
Key Findings
Methodology
MoverScore uses contextual embeddings (e.g., BERT, ELMo) and Earth Mover's Distance (WMD) to compute semantic distance between system outputs and references. It employs multi-layer aggregation techniques like p-means and supports word-level and sentence-level alignment.
Key Results
- On WMT17 machine translation tasks, MoverScore achieved a Pearson correlation of 0.743, outperforming supervised metrics like RUSE.
- On TAC2008 and TAC2009 summarization tasks, it matched or exceeded S3best in correlation with human judgments.
- In image captioning tasks, MoverScore excelled in system-level evaluation, particularly on M1 and M2 scores.
Significance
This study significantly improves semantic correlation in text generation evaluation, addressing the limitations of traditional metrics like BLEU and ROUGE, and provides a unified evaluation framework for multiple tasks.
Technical Contribution
Introduced a framework combining contextual embeddings with Earth Mover's Distance for precise semantic alignment; developed an unsupervised evaluation method applicable across diverse text generation tasks; optimized embedding layer aggregation techniques.
Novelty
First to integrate Earth Mover's Distance with deep contextual embeddings for text generation evaluation, overcoming limitations of surface-form-based metrics.
Limitations
- Weak semantic representation for named entities and numbers, affecting dialogue generation tasks.
- High computational complexity, especially on large datasets.
- Fine-tuning requirements may limit generalizability across tasks.
Future Work
Future work could explore more efficient embedding computation methods, improve representation of named entities and numbers, and extend to tasks like open-domain QA.
AI Executive Summary
Evaluation metrics are critical for advancing text generation systems. However, traditional metrics like BLEU and ROUGE rely heavily on surface-form overlaps, failing to capture semantic correlation.
MoverScore combines contextual embeddings (e.g., BERT, ELMo) with Earth Mover's Distance (WMD) to compute semantic distance between system outputs and references. It achieves high correlation with human judgments across tasks like machine translation, summarization, and image captioning, outperforming several benchmarks.
This study introduces an unsupervised unified evaluation framework and optimizes embedding layer aggregation techniques, significantly enhancing semantic alignment capabilities. Despite limitations in named entity representation and computational efficiency, MoverScore provides a promising direction for future text generation evaluation research.
Deep Analysis
Background
Text generation tasks like machine translation, summarization, and image captioning have traditionally relied on metrics like BLEU and ROUGE, which focus on n-gram overlaps and fail to measure semantic correlation. Recent advances in contextual embeddings (e.g., BERT, ELMo) offer new possibilities for semantic representation.
Core Problem
Existing metrics overly depend on surface forms, failing to accurately reflect semantic consistency between system outputs and references. This hampers fair comparisons across systems.
Innovation
MoverScore innovatively combines contextual embeddings with Earth Mover's Distance, supporting word-level and sentence-level semantic alignment. It uses p-means aggregation to consolidate multi-layer embedding information, enhancing robustness in semantic representation.
Methodology
- �� Generate contextual embeddings using BERT and ELMo.
- �� Compute semantic distance using Earth Mover's Distance.
- �� Apply p-means aggregation to consolidate multi-layer embedding information.
- �� Support word-level and sentence-level alignment for diverse tasks.
Experiments
Evaluated on WMT17 machine translation, TAC2008/2009 summarization, and MS-COCO image captioning tasks. Metrics include Pearson and Spearman correlations with human judgments. Baselines include BLEU, ROUGE, and supervised metrics like RUSE.
Results
MoverScore achieved a Pearson correlation of 0.743 on WMT17 machine translation, outperforming RUSE; matched or exceeded S3best on TAC2008/2009 summarization; excelled in MS-COCO image captioning tasks.
Applications
Applicable to machine translation, summarization, and image captioning tasks, particularly where semantic correlation evaluation is critical.
Limitations & Outlook
Weak representation for named entities and numbers; high computational complexity; fine-tuning requirements may limit generalizability.
Plain Language Accessible to non-experts
Imagine you're comparing two essays to see if they convey the same idea. Traditional methods only check if the words overlap, like checking spelling. MoverScore, however, compares the overall meaning, like checking if the themes of two paintings match.
ELI14 Explained like you're 14
Imagine you're writing an essay, and your teacher wants to know if your essay expresses the same idea as the reference essay. Traditional methods are like checking for spelling mistakes, but MoverScore checks if your essay conveys the same theme and feeling. Cool, right?
Glossary
Earth Mover's Distance
Measures the minimum transportation cost between two distributions, used to compare semantic distances.
Used to compute semantic differences between system outputs and references.
Contextualized Embeddings
Dynamic word vectors generated by deep models, capturing contextual information.
Used to represent semantic information in texts.
p-means
A pooling technique that computes nonlinear averages across embedding layers.
Used to aggregate multi-layer embedding information.
BERT
A Transformer-based language model generating deep contextual embeddings.
Used to generate embeddings for semantic distance computation.
ROUGE
A metric based on n-gram overlaps for text evaluation.
A baseline for summarization tasks.
Open Questions Unanswered questions from this research
- 1 How to improve semantic representation for named entities and numbers?
- 2 How to reduce computational complexity for large-scale datasets?
Applications
Immediate Applications
Machine Translation Evaluation
Helps developers accurately assess the semantic quality of translation systems.
Summarization Optimization
Provides fairer evaluation metrics for summarization models.
Long-term Vision
Open-Domain QA
Extends to complex tasks like open-domain QA, enhancing semantic understanding capabilities.
Abstract
A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.