Evaluation of Text Generation: A Survey

The evolution of NLG has transitioned from rule-based systems to deep neural models, driven by the need for more flexible and context-aware text generation. Early systems relied on handcrafted templates, which limited diversity and scalability. The advent of statistical methods introduced n-gram overlap metrics like BLEU and ROUGE, enabling automatic evaluation but with limitations in capturing semantic meaning. The rise of deep learning, especially transformer architectures (Vaswani et al., 2017), and pre-trained models (GPT, BERT) has significantly improved output quality. Correspondingly, evaluation methods have shifted from surface-level string matching to semantic similarity measures, aiming for more human-like assessments.

Despite technological advances, evaluating NLG remains challenging. Human judgments are costly, subjective, and inconsistent, especially across different tasks. Automatic metrics often fail to correlate well with human perceptions, particularly for creative, diverse, or long-form texts. Existing metrics like BLEU and ROUGE are limited to surface-level overlap, ignoring semantic nuances. There is a pressing need for evaluation frameworks that are scalable, objective, and aligned with human preferences, capable of assessing multiple qualities such as relevance, coherence, diversity, and factual correctness.

The paper proposes a multi-metric evaluation framework that integrates traditional string-based metrics with semantic similarity measures derived from pre-trained models. It introduces fusion strategies—weighted ensembles and machine learning regressors—that enhance correlation with human judgments. The approach emphasizes task-specific adaptation, multi-scale analysis, and robustness across diverse datasets. This comprehensive framework surpasses existing single-metric evaluations, offering a more nuanced and reliable assessment of NLG outputs, especially for complex tasks like long-text generation and dialogue.

• �� Collect diverse datasets (CNN/DailyMail, DSTC, etc.) for summarization, dialogue, and long-text tasks.
• �� Apply traditional metrics (BLEU, ROUGE) for baseline comparison.
• �� Calculate semantic similarity using BERTScore, leveraging contextual embeddings.
• �� Fine-tune BLEURT with domain-specific data for improved factuality assessment.
• �� Develop fusion models combining multiple metrics via weighted averaging and machine learning regressors.
• �� Validate metrics by correlating scores with human judgments (Pearson, Spearman coefficients).
• �� Conduct ablation studies to evaluate contribution of each metric component.
• �� Test across multiple tasks to ensure generalizability and robustness.

Experiments utilize datasets like CNN/DailyMail for summarization, DSTC for dialogue, and custom long-text corpora. Baselines include BLEU, ROUGE, and newer metrics like BERTScore and BLEURT. Metrics are evaluated based on correlation with human ratings, using Pearson and Spearman coefficients. Hyperparameters include learning rate 0.001, batch size 32. Ablation studies compare single metrics versus fused models. Results demonstrate that combined metrics achieve higher correlation (up to 0.65) and better task-specific performance, validating the effectiveness of the proposed framework.

The fusion of BERTScore and BLEURT yields correlation scores of 0.65 in summarization and dialogue tasks, outperforming BLEU (0.45) and ROUGE (0.50). In long-text generation, multi-metric approaches better capture diversity and coherence. Ablation shows that removing semantic metrics reduces correlation by 15-20%. The combined evaluation framework consistently improves model ranking stability and aligns more closely with human judgments, providing a reliable tool for model development.

This evaluation framework can be applied in research to benchmark models, in industry for quality assurance, and in deployment to monitor system outputs. It supports tasks like summarization, chatbots, and content creation, enabling developers to optimize models based on comprehensive, human-aligned metrics. Its adaptability allows integration into automated pipelines, reducing reliance on costly human assessments.

Current metrics depend heavily on pre-trained models, which require significant computational resources. They may still struggle with evaluating creativity, diversity, and factual accuracy comprehensively. Cross-lingual applicability remains limited, and task-specific tuning is often necessary. Further, the fusion strategies may introduce complexity, and interpretability of scores can be challenging. Future work should focus on reducing computational costs and enhancing explainability.