Towards a Similarity-adjusted Surprisal Theory
Proposes similarity-adjusted surprisal, leveraging Ricotta and Szeidl’s diversity index, to enhance reading time prediction beyond standard surprisal.
Key Findings
Methodology
This work extends surprisal theory by integrating Ricotta and Szeidl’s (2006) diversity index to define a similarity-adjusted surprisal metric. Using GPT-2 to estimate word probabilities, the authors incorporate various similarity functions—such as cosine similarity of word embeddings, part-of-speech tags, and orthographic distances—to quantify word relatedness. The mathematical derivation reveals a monotonic relationship between this new measure and information value, bridging the gap between traditional surprisal and communicative equivalences. Empirical validation on datasets like Natural Stories and Dundee demonstrates that this metric improves the prediction of reading times, especially when considering shallow semantic similarities.
Key Results
- In the Natural Stories dataset, adding similarity-adjusted surprisal increased model log-likelihood by approximately 0.5% (∆L=0.0052), outperforming standard surprisal. Dundee results showed non-contextual embedding similarities contributed most, with a 0.35% improvement. For Provo and Brown datasets, the gains were smaller, indicating dataset-specific effects. The experiments confirmed a monotonic relation between surprisal and information value, with similarity functions capturing semantic and orthographic relations that traditional surprisal neglects. These improvements suggest that human language processing considers not only word probabilities but also their relatedness, especially in complex or low-frequency contexts.
- Further analysis revealed that non-contextual semantic similarities yielded better predictive power than contextual ones, implying that shallow semantic processing plays a significant role in incremental comprehension. The mathematical relationship established supports the idea that similarity-adjusted surprisal generalizes standard surprisal, providing a more nuanced measure aligned with cognitive processes. The results highlight the importance of incorporating word relationships into models of language understanding, with potential applications in NLP tasks such as language modeling, machine translation, and cognitive modeling.
- Overall, the findings demonstrate that considering word similarity enhances the predictive accuracy of reading time models, especially in longer texts with complex structures. The approach offers a theoretical framework that unifies surprisal and information value, opening avenues for integrating neural and behavioral data. The study also suggests that shallow semantic cues are more influential than deep contextual integration in certain reading scenarios, aligning with theories of heuristic processing. These insights could inform future research on language comprehension, neural correlates, and computational linguistics.
Significance
This research advances the theoretical understanding of language processing by integrating word similarity into surprisal, a core concept in psycholinguistics. It provides a mathematically grounded framework linking surprisal and information value, addressing limitations of models that treat words as discrete, unrelated entities. Empirically, the enhanced predictive power for reading times demonstrates its potential to improve cognitive models and NLP systems. The approach aligns with cognitive theories emphasizing shallow semantic cues and heuristic processing, offering a more realistic depiction of human language comprehension. Its implications extend to neural studies, where the metric could correlate with brain signals like N400, fostering interdisciplinary progress. Overall, this work bridges theoretical, computational, and experimental domains, enriching our understanding of the cognitive mechanisms underlying language understanding.
Technical Contribution
The paper introduces a formal definition of similarity-adjusted surprisal based on Ricotta and Szeidl’s diversity index, establishing a mathematical relationship with information value. It innovatively incorporates multiple similarity functions—cosine similarity of embeddings, POS tags, orthographic distances—into surprisal calculations, broadening the scope of traditional models. The derivation proves a monotonic, strictly increasing relation between this new measure and information value, providing a unified framework that generalizes standard surprisal. The methodology combines probabilistic language modeling with diversity metrics, enabling nuanced predictions of cognitive effort. The work also offers a theoretical basis for future neural and behavioral studies, suggesting that language comprehension involves graded, similarity-sensitive processes rather than purely probabilistic ones.
Novelty
This is the first systematic integration of diversity indices into surprisal theory, explicitly modeling word similarities rather than treating words as independent outcomes. The novel mathematical formulation bridges surprisal and information value, providing a richer, more cognitively plausible measure of predictability. Unlike prior work focusing solely on probability distributions, this approach captures the continuum of word relatedness, aligning better with human semantic processing. The use of multiple similarity functions and the derivation of a monotonic relationship represent significant theoretical innovations, opening new pathways for modeling language comprehension and prediction in both cognitive science and NLP.
Limitations
- The current model relies heavily on GPT-2 probability estimates, which may not fully capture human prediction mechanisms. The datasets are limited to English, restricting generalizability across languages. The choice of similarity functions influences results significantly, and optimal functions remain to be systematically explored. Computational complexity increases with larger vocabularies and more complex similarity measures, limiting real-time application. Additionally, the model primarily focuses on lexical similarity, neglecting syntactic and discourse-level relations that also impact comprehension. Future work should address these limitations by integrating neural data, expanding language coverage, and optimizing computational efficiency.
Future Work
Future research will explore neural correlates of similarity-adjusted surprisal, validating its relation to brain signals like N400. Extending the framework to multilingual datasets and incorporating syntactic and discourse-level similarities will improve its robustness. Developing more efficient algorithms for real-time estimation and testing in downstream NLP tasks, such as dialogue systems and translation, are also planned. Moreover, integrating dynamic context-sensitive similarity functions could better reflect human incremental processing, advancing both cognitive modeling and practical applications in AI language systems.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家厨房里准备做饭。每次你拿出食材时，都要考虑它们之间的关系，比如番茄和红椒都红色、都可以用来做沙拉。传统的方法就像只看每个食材的重量或颜色，不考虑它们之间的相似性。而这个新方法就像厨师知道，某些食材长得像、味道相似，可以互相替代。这样，你在做菜时，不仅知道每个食材的基本信息，还会考虑它们的相似之处，这样可以更快找到合适的搭配，做出更美味的菜肴。同样，研究者用这个想法，让计算机在理解句子时，不只是看每个词的概率，还会考虑词与词之间的关系。这样，计算机理解句子就更像人类，能更快、更准确地理解复杂的语言内容。
ELI14 Explained like you're 14
想象你在玩拼图游戏，每块拼图代表一个词。有时候，拼图看起来很像，比如颜色和形状都差不多，但其实每块都不一样。以前的电脑模型就像只看每块拼图的编号，不管它们长得像不像。现在，这个新方法就像是告诉电脑：这些拼图有点像，有些颜色和形状很相似。这样，电脑就能更快猜到下一块拼图应该是什么，因为它知道哪些拼图长得像，哪些不像。就像你玩拼图时，如果知道哪些块很相似，就会更快拼好。这项研究用这种想法，让电脑在理解句子时，不只是看每个词的概率，还考虑词与词之间的关系。结果显示，这样做能让电脑更像人一样理解语言，反应得更快、更准确。未来，这种方法还能帮我们做出更聪明的聊天机器人和翻译软件，让它们更懂我们的意思！
Abstract
Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified as surprisal. Traditionally, surprisal theory treats words as distinct entities, overlooking any potential similarity between them. Giulianelli et al. (2023) address this limitation by introducing information value, a measure of predictability designed to account for similarities between communicative units. Our work leverages Ricotta and Szeidl's (2006) diversity index to extend surprisal into a metric that we term similarity-adjusted surprisal, exposing a mathematical relationship between surprisal and information value. Similarity-adjusted surprisal aligns with information value when considering graded similarities and reduces to standard surprisal when words are treated as distinct. Experimental results with reading time data indicate that similarity-adjusted surprisal adds predictive power beyond standard surprisal for certain datasets, suggesting it serves as a complementary measure of comprehension effort.