Neural Network Acceptability Judgments

Recent advances in NLP, driven by models like BERT and GPT, have achieved remarkable success in tasks such as translation, summarization, and question answering. However, these models often lack a true understanding of deep syntactic structures, which are fundamental to human language competence. Traditional linguistic research relies on acceptability judgments by native speakers to study grammar, but scaling this approach computationally has been challenging. Prior datasets, such as those used in the GLUE benchmark, lacked detailed focus on syntactic phenomena. The development of CoLA addresses this gap by providing a large, linguistically curated corpus with expert annotations, enabling systematic evaluation of neural models’ syntactic reasoning. This work builds on earlier efforts like Lau et al. (2016), extending the scope and scale to better reflect linguistic complexity.

Despite the success of neural models in many NLP tasks, their ability to accurately judge sentence grammaticality remains limited, especially for complex structures involving long-distance dependencies and agreement. Existing datasets and models often rely on shallow heuristics or surface cues, failing to capture the underlying syntactic principles. This gap hampers the development of truly linguistically informed models capable of deep language understanding. The core challenge is designing models that can learn and generalize complex syntactic rules from limited data, and reliably distinguish acceptable from unacceptable sentences across diverse linguistic phenomena.

The paper introduces several key innovations:

• �� Construction of CoLA, a large-scale, linguistically annotated corpus covering diverse syntactic and semantic phenomena.
• �� Utilization of pre-trained contextual embeddings (ELMo) combined with max pooling to generate rich sentence representations.
• �� Implementation of transfer learning from unsupervised language modeling tasks to improve syntactic judgment.
• �� Multi-source domain evaluation to assess model robustness and generalization.
• �� Detailed error analysis to identify specific syntactic structures where models fail, guiding future improvements.

These innovations collectively push the boundary of neural syntactic modeling, moving beyond shallow heuristics towards a more linguistically grounded approach.

• �� Data collection: Extract sentences from linguistic sources like Baltin (1982), Levin (1993), and Ross (1967), filtering for acceptability and linguistic relevance.
• �� Annotation: Use expert judgments to label sentences as grammatical or ungrammatical, ensuring high-quality data.
• �� Pretraining: Train an LSTM language model on the British National Corpus (BNC) to learn general language representations.
• �� Feature extraction: Generate contextualized word embeddings (ELMo), then apply max pooling over the sequence to produce fixed-length sentence vectors.
• �� Transfer learning: Fine-tune the sentence encoder on a real/fake discrimination task, where the model learns to distinguish authentic sentences from artificially generated ones.
• �� Classification: Use a multilayer perceptron on top of the frozen encoder to predict acceptability, trained on CoLA labels.
• �� Evaluation: Measure accuracy and MCC on in-domain and out-of-domain test sets, perform ablation studies to analyze the contribution of different components.

The models are trained on the in-domain training set (8,551 sentences) and evaluated on a separate test set (530 sentences). Additional out-of-domain data from different sources tests generalization. Hyperparameters are optimized via cross-validation. Baselines include unsupervised models (Lau et al., 2016) and simple bag-of-words classifiers. Evaluation metrics include accuracy and MCC, with particular attention to performance on complex syntactic structures. Ablation studies remove components like contextual embeddings or transfer learning to assess their impact. The experiments also analyze error patterns across phenomena such as agreement, wh-movement, and semantic violations, providing insights into model limitations.

The best model achieves 67.1% accuracy and MCC of 0.697, considerably below human performance (86.1%). Transfer learning yields a 5% accuracy boost, yet models still struggle with non-local dependencies, with error rates over 30%. Error analysis shows models perform well on simple SVO sentences but poorly on structures involving long-distance dependencies, highlighting the need for architectures that better capture deep syntactic relations. Cross-domain evaluation indicates moderate robustness, but performance drops significantly on out-of-domain data, emphasizing domain-specific limitations.

The models can be integrated into grammar checking tools, language learning platforms, and assistive writing software. They serve as foundational components for more advanced NLP systems requiring syntactic awareness, such as semantic parsers and dialogue agents. Moreover, the insights from error analysis can inform the design of linguistically informed neural architectures, fostering more human-like language understanding in AI systems.

Current models show limited capacity to understand complex non-local dependencies, such as agreement and long-distance wh-movement, which are crucial for deep syntactic comprehension. The dataset, while large, cannot encompass all linguistic phenomena, restricting model generalization. Additionally, models rely less on semantic context, which may cause inaccuracies in ambiguous sentences. Future work should focus on integrating richer syntactic and semantic features, exploring transformer-based architectures, and expanding datasets to include more diverse and complex structures.