Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT

The evolution of machine translation (MT) has transitioned from rule-based systems to neural models like Transformer architectures (Vaswani et al., 2017), significantly improving translation fluency and accuracy. Traditional evaluation relied on metrics such as BLEU (Papineni et al., 2002), which measure n-gram overlap but lack error specificity. Error analysis frameworks like MQM (Quality Metrics for Machine Translation) and MeLLANGE have been developed to classify errors into categories such as terminology, syntax, and content transfer, enabling more granular quality assessment (Freitag et al., 2021). However, manual error annotation is costly and subjective, limiting scalability. Recent advances in large language models (LLMs), especially GPT-3 and GPT-4, have demonstrated capabilities in understanding and generating natural language, prompting research into their application for automatic error detection (Kocmi et al., 2023; Lu et al., 2024). This study builds upon these developments, focusing on the specialized domain of LSP translation, which involves domain-specific terminology and complex syntactic structures, posing unique challenges for error detection.

Existing automatic evaluation methods often lack the granularity needed for professional translation quality control, especially in specialized fields like legal, medical, or technical translation. Manual error annotation ensures accuracy but is labor-intensive and subjective. Automated methods based on generic metrics fail to capture domain-specific errors, such as terminological inaccuracies or complex syntactic errors. The challenge lies in designing prompts that effectively guide LLMs like ChatGPT to identify and classify errors according to detailed typologies, especially in high-resource, domain-specific contexts. Moreover, self-assessment by models remains unreliable, limiting their utility for autonomous evaluation. Addressing these issues requires innovative prompt engineering, error typology adaptation, and systematic evaluation to validate model performance in real-world scenarios.

This work introduces a novel approach combining a customized error typology rooted in MQM and MeLLANGE with advanced prompt engineering techniques. Key innovations include:

• �� Developing two prompt versions—detailed and simplified—to assess the impact of information richness.
• �� Employing prompt chaining to facilitate complex reasoning and comprehensive error localization.
• �� Tailoring error categories for the LSP domain, especially terminological and syntactic errors.
• �� Systematic comparison of model performance across different prompts and translation sources.
• �� Quantitative evaluation against expert annotations using precision, recall, and F1 metrics, establishing benchmarks for future research.

These innovations enable more accurate, domain-specific error detection, surpassing prior generic approaches and setting a new standard for AI-assisted translation evaluation.

• �� Define a comprehensive error typology based on MQM and MeLLANGE, covering terminological, syntactic, content transfer, and other errors.
• �� Design two prompts: a detailed version including error definitions and examples, and a simplified version without detailed descriptions.
• �� Implement prompt chaining to sequentially process translation segments, ensuring complete information flow.
• �� Collect a corpus of NLP research abstracts in English, translated into French via DeepL and self-generated methods.
• �� Engage human experts to manually annotate errors, establishing a gold standard.
• �� Use ChatGPT to annotate errors at sentence level, providing outputs with error spans and categories.
• �� Calculate performance metrics—precision, recall, F1—by comparing AI outputs with human annotations.
• �� Analyze the influence of prompt detail level and translation source on detection accuracy.
• �� Conduct ablation studies to determine the contribution of each component, such as prompt length and error definitions.

The experimental setup involved 35 DeepL translations and 25 self-translations, with human annotations identifying 399 and 193 errors respectively. ChatGPT was prompted with two versions: a detailed prompt containing explicit error definitions and examples, and a simplified prompt omitting these details. Performance was evaluated using precision, recall, and F1 scores at the document level, averaged across all samples. Results showed that the detailed prompt achieved a precision of 79.2%, recall of 65.3%, and F1 of 70.7%, correctly labeling 64.1% of errors. The simplified prompt yielded lower performance, confirming the importance of detailed instructions. The model’s error detection capability was consistent across terminological, syntactic, and content errors, but less effective in self-evaluation scenarios. These experiments validated the approach's effectiveness in high-resource, domain-specific contexts and provided insights into prompt optimization.

The detailed prompt enabled ChatGPT to identify approximately 64% of errors with high precision (79.2%) and moderate recall (65.3%) in DeepL translations, closely matching human annotations. Error classification accuracy was about 64.1%, demonstrating the model’s ability to categorize errors effectively. In contrast, performance on self-generated translations was significantly lower, exposing limitations in self-assessment. The results highlight that prompt richness directly correlates with detection accuracy, especially for complex, domain-specific errors. The findings suggest that prompt engineering is crucial for deploying LLMs in professional translation quality control, with potential for further improvements through multi-turn prompts and fine-tuning.

This approach can be integrated into translation management systems for real-time error detection, reducing manual review efforts. It supports translation training by providing automated feedback on error types, helping students and professionals improve their skills. In industry, it can serve as a scalable quality assurance tool, especially in high-stakes fields like legal or medical translation, where accuracy is critical. Additionally, it offers a foundation for developing hybrid human-AI evaluation workflows, combining automated error detection with expert judgment for optimal results.

Current models struggle with accurate self-assessment, especially in complex or multi-error sentences, limiting autonomous evaluation. The approach’s effectiveness depends heavily on prompt quality; poorly designed prompts reduce accuracy. The experiments focus on high-resource language pairs and NLP domain texts, so applicability to other languages or fields remains uncertain. Computational costs and the need for manual error annotations for validation also pose challenges. Future work must address these limitations by enhancing prompt robustness, expanding domain coverage, and integrating adaptive learning techniques.