Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Over the past decade, neural network-based language models have revolutionized NLP, with architectures like LSTM and Transformer (Vaswani et al., 2017) achieving remarkable performance. Early work such as Mikolov et al. (2010) introduced word embeddings, while Hochreiter and Schmidhuber (1997) laid the groundwork for recurrent neural networks. BERT (Devlin et al., 2019) further advanced the field with bidirectional pretraining. Despite these successes, most evaluations focus on English, neglecting the linguistic diversity of other languages. Prior studies (e.g., Gulordava et al., 2018) tested models on limited languages or corpus-based data, lacking systematic cross-linguistic comparison. As multilingual models like mBERT emerge, understanding their capacity for syntax transfer across languages becomes crucial, especially for low-resource and morphologically rich languages.

While neural models excel in English, their ability to learn and transfer syntactic rules across languages remains unclear. Different languages exhibit diverse syntactic phenomena—such as case marking, agreement, and word order—that challenge models’ generalization. Existing evaluation methods lack control over syntactic structures, making it difficult to isolate specific capabilities. Moreover, models trained on multiple languages often show interference, reducing their effectiveness. Addressing these issues requires a systematic, multilingual evaluation framework that can precisely measure syntactic understanding and transfer, providing insights into the models’ linguistic competence and guiding future improvements.

This work introduces CLAMS, a multilingual, controlled syntactic evaluation suite built upon attribute-varying grammars (AVGs). These grammars enable precise sentence generation with systematic variation of syntactic features, ensuring high experimental control. The framework supports five languages—English, French, German, Hebrew, and Russian—covering diverse morphological and syntactic typologies. The study evaluates both monolingual and multilingual models, revealing that current architectures lack effective syntactic transfer, especially in languages with complex morphology or limited resources. The combination of controlled data generation and cross-lingual comparison represents a significant methodological advance, providing a benchmark for future research.

• �� Develop attribute-varying grammars (AVGs) to generate syntactic test sentences with controlled variation of features like agreement and relative clauses.
• �� Collect and preprocess Wikipedia data for each language, creating balanced train/dev/test corpora.
• �� Train monolingual LSTM models using hyperparameters from prior work (van Schijndel et al., 2019), and a concatenated multilingual LSTM model.
• �� Evaluate models by computing sentence probabilities and accuracy on syntactic constructions, averaging over multiple runs.
• �� Use masked language modeling (MLM) predictions for BERT, comparing scores for grammatical and ungrammatical forms.
• �� Analyze performance differences across languages, constructions, and model types to assess transfer and interference effects.

The experiments involved training models on large Wikipedia datasets, with approximately 80 million tokens per language. Hyperparameters such as hidden units (800), learning rate (20.0), and dropout (0.2) were used. Monolingual LSTMs were trained separately per language, while a combined multilingual LSTM was trained on concatenated data. Evaluation focused on syntactic dependencies like simple agreement, VP coordination, and relative clauses, using sentence probability comparisons for LSTMs and masked token prediction scores for BERT. Each model was tested over five random initializations to ensure robustness. The test sets, generated via AVGs, provided controlled, comparable data across languages and constructions.

Monolingual LSTMs achieved near-perfect accuracy (>99%) on simple dependencies but struggled with nested structures, especially in object relative clauses (~55-60%). Multilingual LSTMs underperformed monolinguals by 3-5%, indicating limited transfer. Multilingual BERT scored 88% in English but only 55-60% in Hebrew and Russian, highlighting resource and script limitations. Languages with richer morphology, like German, showed higher accuracy (~90%), supporting the hypothesis that morphological cues aid syntax learning. These results underscore the challenge of cross-lingual transfer and the importance of language-specific features.

The CLAMS framework can be used to benchmark and improve multilingual language models, guiding architecture and training strategies for better syntactic transfer. Industry applications include multilingual translation, voice assistants, and grammar checking tools, especially for low-resource languages. The controlled evaluation approach aids in diagnosing model weaknesses, informing targeted improvements. Long-term, this research supports the development of universal NLP systems capable of understanding and generating syntactically correct sentences across diverse languages, fostering more inclusive AI technologies.

The current test set focuses on specific syntactic phenomena, leaving out others like negation or coordination. Model hyperparameters were not optimized per language, possibly affecting performance. Vocabulary coverage issues, especially for low-resource languages, limit BERT’s effectiveness. The evaluation relies on specific constructions, which may not reflect broader syntactic capabilities. Additionally, resource imbalance and script differences pose challenges for low-resource and non-Latin languages, requiring further adaptation and data collection for comprehensive assessment.