SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
Multi-model ensemble approach combining feature engineering and deep learning achieved an average Pearson correlation of 0.73 in multilingual STS tasks.
Key Findings
Methodology
This paper introduces a multilingual Semantic Textual Similarity (STS) framework that integrates feature engineering—such as n-gram overlap, edit distance, word alignment, MT evaluation metrics (BLEU, METEOR)—with deep neural models like LSTM and CNN. Multiple models, including Random Forest, Gradient Boosting, and XGBoost, are trained on features extracted from datasets in English, Arabic, Spanish, and Turkish, with machine translation used for cross-lingual pairs. The ensemble combines outputs via weighted averaging and stacking, enhancing robustness. The system is evaluated on SemEval-2017 datasets, with performance measured by Pearson correlation against human annotations.
Key Results
- The ensemble system achieved an average Pearson correlation of 0.73 across all tracks, with the highest in Spanish (0.86) and Turkish-English (0.77). Cross-lingual performance dropped by over 10%, but the models maintained strong robustness through feature fusion and ensemble strategies, outperforming baseline models (correlation 0.54). The results demonstrate that combining symbolic features with neural embeddings significantly improves multilingual semantic matching.
- In particular, the models showed consistent performance in low-resource languages, indicating the effectiveness of translation-based data augmentation and multi-source feature integration. Ablation studies confirmed that feature fusion contributed approximately 15% to the overall performance gain.
- The models excelled in both monolingual and cross-lingual tasks, with ensemble strategies mitigating language-specific challenges. The results suggest that multi-model fusion is essential for scalable, accurate multilingual semantic similarity assessment in real-world applications.
Significance
This work advances the state-of-the-art in multilingual semantic similarity evaluation, addressing key challenges such as language resource disparity and semantic divergence. By effectively combining symbolic and distributed representations, the approach enhances the understanding of cross-lingual semantics, which is vital for applications like multilingual search engines, cross-lingual question answering, and machine translation evaluation. The methodology offers a scalable, adaptable framework that can be extended to additional languages and domains, promoting more inclusive NLP systems. Its success demonstrates the potential of model ensembling and feature fusion in overcoming resource limitations and improving semantic comprehension across languages.
Technical Contribution
The paper's main contribution lies in the innovative integration of diverse features—lexical, syntactic, and semantic—within a multi-model ensemble architecture. It introduces a systematic approach to combining symbolic features (e.g., n-grams, edit distance, word alignment) with neural embeddings (e.g., GloVe, sentence vectors) across multiple models (RF, GB, XGBoost, LSTM, CNN). The ensemble strategy, including stacking and weighted averaging, significantly boosts performance over single models. Additionally, the use of machine translation for cross-lingual data augmentation and the comprehensive evaluation across multiple languages set new standards for multilingual STS systems, providing both theoretical insights and practical tools for NLP research.
Novelty
This research is the first to systematically combine feature engineering and deep neural models within a multi-model ensemble for multilingual STS, especially in low-resource scenarios. Unlike prior work focusing solely on monolingual models or shallow features, this approach leverages the complementary strengths of symbolic and distributed representations, achieving superior robustness and accuracy. The innovative use of translation-based data augmentation and multi-source feature fusion distinguishes it from existing methods, marking a significant step forward in scalable, cross-lingual semantic understanding.
Limitations
- Despite strong results, the model's performance declines in low-resource languages due to limited training data and translation quality issues, which can introduce noise and reduce accuracy.
- The feature engineering process is complex and computationally intensive, requiring extensive tuning and domain expertise, limiting scalability.
- The approach relies heavily on machine translation quality; errors in translation can propagate and affect semantic similarity scores, especially in languages with less mature MT systems.
Future Work
Future research will focus on integrating large-scale pre-trained multilingual models like mBERT and XLM-R to reduce dependency on feature engineering. Exploring end-to-end neural architectures that learn semantic similarity directly from raw text, combined with multi-task learning, can further improve robustness. Additionally, expanding datasets for low-resource languages and refining translation quality will be prioritized. Incorporating contextual and reasoning capabilities into models may also address current limitations in handling complex semantic relations, paving the way for more universal multilingual NLP systems.
AI Executive Summary
Semantic Textual Similarity (STS) plays a crucial role in natural language understanding, enabling machines to assess how closely two sentences express the same meaning. As global communication expands, the demand for accurate multilingual semantic matching grows. Traditional approaches relied heavily on lexical features or shallow models, which often failed to capture the nuanced semantic relationships across languages. Recent advances in deep learning have improved performance, but challenges remain, particularly in low-resource languages and cross-lingual contexts.
This paper presents a comprehensive framework that combines feature engineering with deep neural models, employing an ensemble strategy to enhance multilingual STS performance. The system integrates symbolic features such as n-gram overlap, edit distance, word alignment, and MT evaluation metrics with distributed embeddings like GloVe and sentence vectors. Multiple models—including Random Forest, Gradient Boosting, XGBoost, LSTM, and CNN—are trained on these features, then fused through stacking and weighted averaging. This multi-source, multi-model approach significantly outperforms baseline methods, achieving an average Pearson correlation of 0.73 across diverse datasets.
Experimental results demonstrate the robustness of the ensemble system, with particularly high performance in Spanish (0.86) and Turkish-English (0.77). Although cross-lingual performance drops by over 10%, the models maintain strong generalization, validating the effectiveness of feature fusion and model stacking. The approach effectively addresses resource scarcity issues by leveraging machine translation and multi-source features, making it applicable in real-world multilingual applications such as search engines, question answering, and translation evaluation.
The significance of this work lies in its innovative combination of symbolic and neural features within a scalable ensemble framework, setting new benchmarks for multilingual semantic similarity assessment. It offers a practical, adaptable solution that can be extended to additional languages and domains, fostering more inclusive and intelligent NLP systems. Despite these advances, limitations such as translation quality dependence and computational complexity remain. Future directions include integrating large pre-trained multilingual models, developing end-to-end architectures, and expanding low-resource language datasets, aiming for more accurate, efficient, and universal multilingual semantic understanding.
Deep Dive
Abstract
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).