Extending Automatic Machine Translation Evaluation to Book-Length Documents

Long-form machine translation (MT) has evolved from simple concatenation of sentences to sophisticated models incorporating broader context, such as multi-encoder architectures and large-scale document corpora. Despite these advances, evaluation methodologies lag behind, primarily limited to sentence-level metrics like BLEU, which cannot capture discourse coherence or content omissions in lengthy texts. Recent efforts include model-based metrics like COMET and MetricX, trained on segment-level data, but they struggle with long documents due to token limits and rigid sentence boundary assumptions. Manual evaluation remains costly and subjective, prompting the need for automated, robust, and scalable evaluation frameworks capable of handling entire books or lengthy articles.

The core challenge lies in reliably assessing translation quality over arbitrary-length documents without being constrained by token limits or sentence boundary dependencies. Existing metrics cannot effectively detect content omissions, hallucinations, or boundary inconsistencies in long texts. Additionally, many models are tested only within their maximum token capacities, which are often far below real-world long documents. This disconnect hampers progress in developing truly long-context-aware translation systems, as there is no unified, automated evaluation method that can handle the complexities of book-length translations, including varied sentence boundaries and content integrity issues.

SEGALE introduces a comprehensive solution: • Sentence segmentation using off-the-shelf tools to handle diverse sentence boundaries. • Dynamic sentence alignment via Vecalign with adaptive skip costs, optimized through heuristic search, to accurately match source and translation segments. • Custom semantic embedding models trained on synthetic data to improve alignment robustness across languages and boundary variations. • Null alignment detection to identify and penalize content omissions and hallucinations, ensuring content completeness. • Aggregation of sentence-level scores into a holistic document-level evaluation, compatible with existing metrics like COMET and MetricX. These components collectively enable reliable, scalable evaluation of long documents, overcoming token length constraints and boundary rigidity.

The experimental setup involves datasets from WMT 2024, covering multiple language pairs and including human judgments for correlation analysis. The authors simulate translation errors by removing sentences or merging neighboring sentences, creating scenarios of over- and under-translation. They compare SEGALE against baselines such as using true sentence boundaries (groundtruth) and the mwerSegmenter alignment scheme. Evaluation metrics include Kendall’s τ correlation with human scores and null alignment ratios. Multiple ablation studies test different sentence embeddings and segmentation tools, confirming the robustness of the proposed framework across scenarios. Results demonstrate superior correlation with human judgments, especially in error-prone and boundary-variant cases.

SEGALE achieves Kendall’s τ scores around 0.3085, close to the groundtruth (0.3110), outperforming mwerSegmenter (0.2874). It maintains high robustness in scenarios with boundary variations and content omissions, with error rates rising sharply in models exceeding their context window. The custom BGE-M3 embedding significantly improves alignment accuracy over LASER and original BGE-M3. In book-length evaluation, the framework detects translation quality degradation when models surpass their maximum context length, validating its practical utility. Overall, the results confirm that SEGALE provides a reliable, scalable, and precise long-document evaluation method, bridging the gap between model capabilities and evaluation needs.

This framework can be directly applied to evaluate long texts such as books, legal documents, and research papers, facilitating quality control in industrial translation pipelines. It supports multi-language, multi-domain scenarios, enabling developers to optimize models for discourse-level coherence. In academia, it aids in benchmarking long-text translation systems and analyzing model limitations. Long-term, integrating SEGALE with real-time translation systems and multi-modal content analysis could revolutionize comprehensive content understanding, enabling AI to process and evaluate entire books or extensive reports efficiently.

Despite its robustness, SEGALE still faces challenges in alignment accuracy when paraphrasing or complex sentence structures are present. Null alignment detection may misclassify semantically similar but content-different sentences, affecting stability. The computational cost increases with document length, requiring further optimization for ultra-long texts. Future work should focus on improving alignment algorithms, reducing processing time, and extending multi-modal evaluation capabilities to handle images and audio embedded in long documents.