A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles

Recent advances in deep learning, especially Transformer-based models like BERT, GPT, and RoBERTa, have significantly improved NLP tasks. Psycholinguistic studies have shown that humans process argument roles with a delay, integrating structural information dynamically. Prior research has explored models’ ability to handle syntax, but often focusing on lexical cues or sentence-level metrics. Recent efforts have used probing and surprisal measures to evaluate structural sensitivity, yet a comprehensive understanding remains elusive. This study builds on these foundations, aiming to systematically compare models’ internal representations with human sentence processing, particularly regarding argument roles, a core aspect of syntactic comprehension.

Despite high performance on many NLP benchmarks, current models struggle with understanding and generalizing argument structures, especially in manipulated sentences where roles are reversed or verbs change. This gap affects their interpretability and robustness, limiting real-world applications like translation, question answering, and dialogue systems. The core challenge is whether models encode structural relations explicitly or rely on surface cues, and how this impacts their ability to mimic human-like processing. Addressing this requires detailed, multi-faceted evaluation methods that can dissect internal representations and behavioral responses, revealing the mechanisms underlying structural understanding.

The main innovations include: 1) adopting psycholinguistic experimental paradigms—minimal pairs, neural and behavioral metrics—to evaluate structural sensitivity systematically; 2) integrating surprisal, layer probing, and attention analysis to dissect how models encode argument roles across layers; 3) designing controlled manipulations (argument swap, verb change, argument replacement) to isolate structural effects from lexical cues; 4) comparing unidirectional and bidirectional models, revealing differences in internal encoding and processing. This comprehensive framework advances the understanding of how neural models process syntactic relations, bridging cognitive science and AI.

• �� Construct minimal sentence pairs with controlled argument manipulations, including swap-arguments, change-verb, and replace-argument conditions. • Compute surprisal values for target verbs, analyzing differences between plausible and implausible contexts. • Use layer-wise probing classifiers trained on verb representations to assess the encoding of argument roles at each layer. • Perform attention analysis to identify whether models correctly focus on subjects and objects, indicating structural recognition. • Compare neural responses with human electrophysiological data (N400 amplitudes) to validate cognitive plausibility. • Evaluate multiple models (GPT-2, BERT, RoBERTa) across different sizes, analyzing how architecture influences structural sensitivity.

The experiments involve three key analyses: 1) surprisal measurement to quantify prediction difficulty across conditions, revealing models’ sensitivity to argument plausibility; 2) layer probing to analyze where in the network argument information is encoded, with high accuracy indicating strong internal representation; 3) attention analysis to verify whether models correctly allocate focus to syntactic roles, especially subjects. The datasets are based on psycholinguistic stimuli from Chow et al. (2016) and Kim et al. (2005), with balanced conditions to isolate structural effects. Results are compared across models and layers, providing a comprehensive picture of structural encoding and its limitations.

Models show partial sensitivity: surprisal differences in change-verb conditions exceed 1.2 bits with >85% accuracy, but sensitivity drops sharply in swap-arguments, with GPT-2 small near chance. Layer probing indicates middle layers encode argument roles more robustly than final layers, especially in RoBERTa. Attention analysis confirms correct focus on subjects but limited structural integration. Overall, models rely on lexical cues and surface features, with weaker dynamic structural understanding compared to humans. These findings highlight the need for architecture improvements to better capture syntactic relations.

The evaluation framework can be used to benchmark and improve NLP systems in tasks requiring deep syntactic understanding, such as machine translation, question answering, and dialogue systems. Incorporating explicit structural modules or multi-modal data could enhance robustness and interpretability. The insights also inform cognitive modeling, aiding the development of AI that mimics human sentence processing more closely. Practical deployment in real-world applications demands models with stronger generalization to complex syntactic phenomena, which this research aims to facilitate.

The current study focuses on English, limiting cross-linguistic generalization. Models show weak sensitivity in complex or nested structures, and reliance on surface cues restricts their interpretability. Computational costs of probing and attention analysis are high, hindering scalability. Future work should explore integrating explicit syntactic representations, multi-modal training, and cross-lingual evaluations to address these limitations, moving toward more cognitively plausible models.