AskQE: Question Answering as Automatic Evaluation for Machine Translation
AskQE employs a QA framework with LLaMA-3 70B, achieving high correlation (τ=0.878) with human judgments for critical MT error detection.
Key Findings
Methodology
This study introduces AskQE, a question generation and answering framework that assesses MT quality by extracting facts from source sentences, applying natural language inference (NLI) to filter reliable facts, and generating questions using LLAMA-3 70B. The system answers questions based on both source and backtranslated MT outputs, comparing answer consistency to detect errors. Synthetic high-risk errors are simulated via CONTRATICO, and model configurations are optimized through experiments. The final system demonstrates superior performance on BioMQM, with a Kendall's τ of 0.878 and decision accuracy surpassing traditional QE metrics.
Key Results
- AskQE effectively distinguishes minor from critical errors, with an average F1 of 0.815 and EM of 0.682 on CONTRATICO, across multiple language pairs. It shows high sensitivity to error severity, with scores decreasing as errors become more impactful.
- On the natural error BioMQM dataset, AskQE correlates strongly with human judgments (τ=0.878), outperforming XCOMET-QE and BT-Score, indicating its robustness and interpretability.
- Decision experiments reveal that AskQE-based feedback significantly improves high-stakes decision accuracy, demonstrating practical utility in real-world applications.
Significance
This work advances MT quality estimation by integrating explainable question-answering mechanisms, addressing the need for intuitive, actionable feedback for monolingual users in critical contexts. It overcomes limitations of scalar scores and opaque error annotations, providing detailed explanations aligned with human judgment. The approach enhances trust and reliability in automated translation systems, especially in healthcare and legal domains, fostering safer deployment of MT technology.
Technical Contribution
The paper innovatively combines fact extraction, natural language inference, and large language models for question generation, creating a novel QA-based evaluation pipeline. It leverages LLAMA-3 70B for high-quality question generation and employs multiple similarity metrics for answer comparison, establishing a new paradigm for explainable MT quality assessment. The method’s sensitivity to error severity and its ability to produce interpretable feedback mark significant progress over existing metrics.
Novelty
This is the first work to embed question-answering into MT quality estimation at a fine-grained, sentence-level, utilizing factual extraction and entailment to guide question generation. Unlike prior approaches relying solely on reference-based metrics or error span annotations, AskQE’s explainable framework offers a new perspective on error detection, especially in high-stakes scenarios.
Limitations
- The system's accuracy diminishes with complex semantic errors or ambiguous expressions, where factual extraction and entailment may be less reliable, potentially leading to missed errors.
- Dependence on large-scale models like LLAMA-3 70B incurs high computational costs, limiting real-time deployment without optimization.
- Backtranslation introduces noise, which can affect answer consistency, necessitating further research into noise reduction and multi-modal integration.
Future Work
Future directions include expanding multilingual capabilities, integrating multimodal data (e.g., images, context), and enhancing model efficiency. Exploring active learning and user-in-the-loop approaches could improve question quality and interpretability. Broader application in legal, financial, and safety-critical domains is also envisioned, aiming to develop comprehensive, scalable, and trustworthy MT evaluation systems.
AI Executive Summary
Machine translation (MT) has become indispensable in bridging language barriers across numerous sectors. However, in high-stakes environments like healthcare, legal, and emergency communication, translation errors can have severe consequences. Existing quality estimation (QE) metrics, such as scalar scores or error span annotations, often lack interpretability and fail to provide actionable insights for monolingual users who cannot assess the target language directly.
This challenge motivates the development of AskQE, an innovative framework that leverages question answering (QA) to evaluate translation quality. The core idea is to generate questions based on the source sentence’s factual content, then answer these questions using both the source and the backtranslated MT output. Discrepancies in answers serve as indicators of critical errors, enabling users to make informed accept/reject decisions. The system employs a combination of fact extraction, natural language inference (NLI), and large language models (LLAMA-3 70B) for question generation, ensuring high relevance and quality.
Extensive experiments on synthetic high-risk error datasets (CONTRATICO) and real-world natural error datasets (BioMQM) demonstrate the effectiveness of AskQE. On CONTRATICO, it distinguishes minor from critical errors with an F1 of 0.815, and on BioMQM, it correlates strongly with human judgments (τ=0.878). These results surpass traditional QE metrics, highlighting the potential of QA-based evaluation for practical deployment.
The approach offers significant implications for the deployment of MT in sensitive fields, providing transparent, explainable, and actionable feedback. It addresses key limitations of existing metrics by focusing on interpretability and error severity sensitivity, paving the way for safer, more reliable machine translation systems. Future work will focus on multilingual expansion, computational efficiency, and integration into real-time decision-making workflows, aiming to transform high-stakes translation quality assessment.
Deep Dive
Abstract
How can a monolingual English speaker determine whether an automatic translation in French is good enough to be shared? Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. We introduce AskQE, a question generation and answering framework designed to detect critical MT errors and provide actionable feedback, helping users decide whether to accept or reject MT outputs even without the knowledge of the target language. Using ContraTICO, a dataset of contrastive synthetic MT errors in the COVID-19 domain, we explore design choices for AskQE and develop an optimized version relying on LLaMA-3 70B and entailed facts to guide question generation. We evaluate the resulting system on the BioMQM dataset of naturally occurring MT errors, where AskQE has higher Kendall's Tau correlation and decision accuracy with human ratings compared to other QE metrics.