Same evaluation, more tokens: On the effect of input length for machine translation evaluation using Large Language Models

Over the past decade, machine translation evaluation has evolved from simple n-gram overlap metrics like BLEU to more sophisticated, model-based approaches. Large language models (LLMs) such as GPT-4 and Claude have demonstrated remarkable capabilities in predicting translation errors at a fine-grained level, leveraging frameworks like MQM for detailed error span annotations. These models excel in short-form evaluation but face challenges with long documents due to their input length limitations. Prior work attempted to evaluate long texts via segmentation or concatenation, but these methods often led to inconsistent results and reduced reliability. As the demand for high-quality, long-form translation assessment grows—especially in legal, scientific, and technical domains—understanding and mitigating length bias in LLMs has become a critical research frontier.

Despite the success of LLMs in translation evaluation, their performance degrades significantly with increasing input length. This manifests as fewer error spans detected in long documents and a decline in system ranking accuracy—sometimes by over 12%. The core issue stems from the models’ limited context windows and their tendency to focus on local rather than global information when processing lengthy inputs. This length bias hampers the reliability and interpretability of automated evaluation systems, which are vital for ensuring translation quality in real-world applications involving lengthy texts. Addressing this challenge requires strategies that enable models to maintain consistent error detection and ranking accuracy regardless of input size.

The key innovation is the Focus Sentence Prompting (FSP) strategy, which prompts models to evaluate one sentence at a time within the full document context, thus reducing length bias. This approach allows models to focus on local content while still leveraging global information, avoiding the pitfalls of processing excessively long inputs. Additionally, the authors develop length-matched demonstrations (GMICL-5), selecting examples that match the test input length to improve robustness. They also implement a fine-tuning (GMFT) procedure, training models on multi-length data to enhance their adaptability. These combined strategies enable models to detect errors more reliably across different input granularities, significantly improving system ranking accuracy and error span detection in long documents.

• �� Use MQM error span annotations as the evaluation metric, analyzing how input length affects detection accuracy.
• �� Design Focus Sentence Prompting (FSP): provide the full document but prompt the model to evaluate one sentence at a time, maintaining context.
• �� Incorporate length-matched demonstrations (GMICL-5): select examples with similar length to the test input to improve robustness.
• �� Fine-tune models (GMFT) on multi-length data, enabling better generalization across granularities.
• �� Optimize inference efficiency via prompt caching, sharing prompts across evaluations.
• �� Evaluate on WMT datasets, measuring system pairwise accuracy, error span F1, and computational costs across models and languages.

The experiments utilize WMT’24 and WMT’23 datasets, covering multiple language pairs and translation directions. Models tested include GPT-4, Claude 3.5, Qwen 2.5, and DeepSeek V3. The evaluation compares segment, document, and multi-document inputs, assessing error span detection and system ranking accuracy. Baselines include fixed three-shot prompts, with variations incorporating GMICL-5 demonstrations and fine-tuning (GMFT). Metrics include pairwise accuracy, error span F1, and inference time. Ablation studies analyze the impact of prompt design, demonstration matching, and fine-tuning. Results demonstrate that FSP combined with fine-tuning consistently outperforms baseline methods, especially in long-form scenarios.

FSP effectively stabilizes error detection across input granularities, increasing system ranking accuracy by over 12% in long documents. It also reduces the error span detection gap between segment and document levels, improving consistency. Fine-tuning (GMFT) further enhances robustness, maintaining high accuracy and reducing variance. The experimental results confirm that these strategies generalize across models and languages, with minimal inference overhead thanks to prompt caching. The findings validate that focusing on individual sentences within the full context addresses the core length bias problem, leading to more reliable evaluation outcomes.

This approach can be integrated into automated translation evaluation pipelines, especially for long documents such as legal contracts, scientific papers, and technical manuals. It enables industry practitioners to obtain consistent, interpretable quality assessments without manual intervention. The strategies are compatible with existing LLM-based evaluation tools and can be deployed in real-time systems, improving the reliability of long-form translation quality control. Additionally, the methodology can inform future development of length-robust evaluation metrics and tools, fostering broader adoption in industry and academia.

The current study is limited to three language pairs and datasets, necessitating broader validation. FSP involves multiple inference passes, increasing computational costs, which may hinder real-time deployment. The focus on error span detection overlooks other quality aspects like coherence and discourse-level errors. Further research is needed to optimize inference efficiency, extend to more languages, and incorporate document-level phenomena such as coreference and coherence. Additionally, the reliance on annotated MQM data limits applicability in low-resource scenarios.