DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering

The evolution of natural language generation (NLG) has been driven by large-scale pre-trained language models such as GPT, BART, and T5, which have significantly improved fluency and coherence in tasks like summarization and dialogue. Traditional evaluation metrics like BLEU, ROUGE, and METEOR rely on n-gram overlap, which often fails to capture semantic adequacy and content fidelity. Recent model-based metrics, including BERTScore and BARTScore, leverage contextual embeddings but depend on supervised training or pseudo-labeled data, limiting their generalization. UniEval introduced a reformulation as a question-answering task, but still required training on evaluation datasets. The advent of instruction tuning (e.g., InstructGPT, T0) has enabled models to follow natural language instructions zero-shot, inspiring new frameworks for evaluation. This paper builds upon these developments, proposing a zero-shot, interpretable evaluation method that combines instruction tuning with sentence-level question decomposition, aiming to address the limitations of existing metrics.

Current evaluation metrics for NLG suffer from poor generalization across tasks and lack interpretability. Supervised metrics overfit to specific datasets, reducing their applicability to unseen tasks or dimensions. Traditional metrics ignore deep semantic content, leading to unreliable assessments. Moreover, most neural metrics produce a final score without explaining the reasoning behind it, which hampers trust and diagnostic utility. Developing a universal, unsupervised, and explainable evaluation framework remains a critical challenge, especially as NLG applications diversify and demand more nuanced quality assessments.

The core innovation of DecompEval is its formulation of evaluation as an instruction-based question answering task, utilizing instruction-tuned PLMs in a zero-shot manner. It introduces a sentence-level question decomposition strategy, where the overall evaluation question is broken down into subquestions about individual sentences. These subquestions are answered sequentially, with their answers serving as evidence for the final judgment. This approach enhances interpretability by pinpointing specific low-quality segments and improves generalization by avoiding reliance on task-specific training data. The question reassembly mechanism ensures holistic evaluation, making the method adaptable to multiple tasks and dimensions.

• �� Transform the evaluation task into an instruction-style question, including an instruction, evaluation input (context, generated text, reference), and a yes/no question.
• �� Decompose the main question into sentence-level subquestions, each assessing a specific sentence’s quality.
• �� Use instruction-tuned PLMs (e.g., FLAN-T5) to answer each subquestion sequentially, appending each answer to the prompt as in-context evidence.
• �� Collect all subquestion answers, reassemble them with the original question, and compute the final score based on the probability of 'yes' versus 'no' answers.
• �� This process enables multi-dimensional evaluation and provides interpretability by highlighting problematic sentences.

Experiments utilize SummEval and Topical-Chat datasets, covering summarization and dialogue tasks. Baselines include ROUGE, BERTScore, UniEval, and BARTScore. The model employs FLAN-T5-XL (3B parameters), with input length set to 1024 tokens. Evaluation metrics include correlation coefficients (Spearman, Kendall) and task-specific scores. Ablation studies assess the impact of question decomposition and prompt design. Results demonstrate superior correlation scores, with DecompEval outperforming baselines in zero-shot settings, validating its robustness and generalization.

DecompEval achieves a Spearman ρ of 0.455 and Kendall τ of 0.378 on SummEval, surpassing trained models like UniEval (ρ=0.575, τ=0.442) in certain dimensions. In dialogue evaluation, it outperforms traditional metrics and trained models across multiple dimensions, especially in groundedness and coherence. Its sentence-level analysis allows precise identification of low-quality segments, improving interpretability. The results confirm that the method generalizes well across tasks and dimensions, with consistent performance even on unseen evaluation criteria.

This framework can be directly applied to automatic evaluation of summarization, dialogue, and data-to-text generation, especially in scenarios requiring explainability and cross-task adaptability. It facilitates model development, benchmarking, and quality assurance without extensive labeled data. In industry, it supports content moderation, model tuning, and user feedback analysis, enabling scalable and transparent quality control. Its flexibility allows integration into multi-modal systems with minimal modifications.

The approach may face challenges with very long texts or complex sentence structures, where sentence segmentation and context understanding are difficult. Sensitivity to prompt phrasing requires careful design, and computational costs are higher than traditional metrics due to multiple inference steps. Its effectiveness in highly specialized or domain-specific tasks remains to be validated. Future work should focus on improving robustness, efficiency, and multimodal extension to broaden applicability.