Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives
Introduces 'information value', a measure based on neural language models quantifying utterance predictability via multi-dimensional distances, outperforming token surprisal.
Key Findings
Methodology
Using autoregressive neural language models (GPT-2, GPT-Neo), the study samples multiple plausible alternative utterances conditioned on context. It computes lexical, syntactic, and semantic distances between target and alternatives, summarizing these into a distribution representing predictability. Multiple sampling strategies (temperature, nucleus, typical sampling) ensure robustness. The empirical evaluation involves correlating these measures with human acceptability judgments and reading times across diverse datasets, demonstrating superior predictive power over traditional token-level surprisal. The approach offers interpretability by dissecting different cognitive dimensions influencing comprehension.
Key Results
- In dialogue acceptability prediction, the semantic distance-based information value correlates at about -0.7 (Spearman), significantly better than average token surprisal (~-0.5). For reading times, the correlation reaches approximately 0.42, with different distance metrics showing task-specific predictive strengths. The robustness across models and sampling strategies confirms stability. Semantic distances dominate in dialogue, while lexical and syntactic distances better predict reading durations, revealing distinct cognitive mechanisms.
- Across datasets, the proposed measure consistently outperforms traditional perplexity, especially in predicting human judgments. The multi-dimensional approach clarifies how different aspects of predictability contribute to comprehension, providing insights into semantic expectations and syntactic processing. The method's interpretability enables detailed analysis of how language expectations shape human responses.
- Parameter sensitivity analysis shows larger candidate sets improve prediction, while model size has limited impact, indicating the method's scalability. Linear mixed-effects modeling confirms that semantic predictability influences acceptability, whereas lexical and syntactic predictability better explain reading times, highlighting task-specific cognitive processes.
Significance
This work advances the understanding of language predictability by integrating multi-dimensional, interpretable measures grounded in neural language models. It addresses the limitations of token-level surprisal by capturing broader contextual expectations, thus providing a richer framework for modeling human comprehension. The measure's ability to distinguish different cognitive layers offers new avenues for cognitive science, NLP, and AI, facilitating the development of more natural and human-like language systems. Its robustness and interpretability make it a valuable tool for both theoretical exploration and practical application in dialogue systems, reading comprehension, and cognitive modeling.
Technical Contribution
The paper introduces a novel framework combining neural sampling, multi-distance metrics, and distributional summaries to quantify utterance predictability at the full-sentence level. Unlike traditional token surprisal, this approach models the entire utterance space, capturing semantic, syntactic, and lexical expectations simultaneously. It leverages interpretable distance metrics and empirical sampling to produce stable, explainable estimates. The methodology is compatible with existing neural language models and sampling strategies, offering a scalable, robust, and interpretable alternative for language modeling and cognitive analysis. It also provides a new perspective on how to incorporate multi-dimensional expectations into computational models of language processing.
Novelty
This study is the first to operationalize a full-utterance predictability measure based on neural model sampling, explicitly modeling the distance from a set of plausible alternatives across multiple linguistic dimensions. Unlike token-level surprisal, which is limited to word prediction probabilities, the 'information value' captures broader contextual expectations, enabling detailed interpretability. Its multi-distance, distribution-based approach offers a new paradigm for understanding human language comprehension and improving language models' alignment with cognitive processes.
Limitations
- The candidate set generation depends heavily on the quality of neural models, which may introduce biases or inaccuracies, especially in low-resource languages or domains. Computational costs are high due to sampling and distance calculations over large sets, limiting real-time application. The distance metrics, while interpretable, may not fully capture the complexity of human expectations, especially in nuanced pragmatic or emotional contexts. Future work should focus on optimizing efficiency and expanding the range of cognitive dimensions modeled.
Future Work
Future research will explore integrating multimodal cues (e.g., prosody, gestures) into the predictability measure, aiming to better model real-world language understanding. Extending the framework to multilingual settings and more diverse discourse contexts will test its universality. Additionally, optimizing sampling algorithms and distance computations will improve scalability. The approach could also inform the design of more human-like dialogue agents and enhance neurocognitive models of language processing, fostering deeper insights into the mechanisms of human comprehension.
AI Executive Summary
Understanding how humans predict and interpret language remains a central challenge in cognitive science and artificial intelligence. Traditional measures like token-level surprisal, while useful, often fall short in capturing the full complexity of human expectations, especially across different linguistic dimensions such as semantics, syntax, and lexical choice. This limitation hampers the development of models that truly mirror human comprehension and hampers the interpretability of computational predictions.
In response, this study introduces 'information value,' a novel measure rooted in neural language models. By sampling plausible alternative utterances conditioned on context, and measuring their distances from the target across multiple dimensions, the authors construct a comprehensive predictability metric. This approach leverages the strengths of large-scale models like GPT-2 and GPT-Neo, employing strategies such as temperature and nucleus sampling to generate diverse, plausible alternatives. The core innovation lies in quantifying how far a target utterance is from these alternatives, thus capturing its unexpectedness in a way that aligns more closely with human cognition.
Empirical validation across datasets such as SWITCHBOARD, DAILYDIALOG, PROVO, and BROWN demonstrates that information value correlates more strongly with human judgments of acceptability and reading times than traditional token surprisal. Notably, semantic distances dominate in dialogue acceptability, while lexical and syntactic distances better predict reading durations, revealing distinct cognitive processes. The robustness of the measure across different models and sampling strategies further underscores its potential as a reliable tool for linguistic and cognitive analysis.
This work offers significant implications for both theory and application. It provides a more nuanced understanding of language predictability, facilitating the development of more natural and human-like AI systems. Its interpretability enables detailed analysis of the cognitive dimensions involved in language comprehension, opening new avenues for research in neurocognition, NLP, and AI. Despite computational costs and reliance on model quality, the approach sets a foundation for future innovations in modeling human language understanding, with promising directions including multimodal integration and multilingual extensions.
Deep Dive
Glossary
Surprisal (困惑度)
衡量语言单位在给定上下文中出现的意外程度，基于信息熵的负对数概率。技术上，I(u) = -log2 p(u)。
用来评估一句话的预测难度和认知负荷。
信息价值 (Information Value)
衡量一句话相对于候选集的距离，反映其预测性和出乎意料的程度，结合多维距离指标。技术上是话语空间中距离的分布。
本文提出的新指标，用于更全面描述话语的预测性。
距离指标 (Distance Metrics)
包括词汇（n-gram差异）、句法（POS n-gram差异）和语义（句子嵌入距离），用于衡量话语间的差异。
用以计算目标话语与候选集的差异，反映不同认知层面。
候选集 (Alternative Set)
在特定上下文中由神经模型采样生成的多个合理话语候选，用以衡量目标话语的预测性。
核心组成部分，用于计算信息价值。
采样策略 (Sampling Strategies)
包括温度采样、核采样、典型采样等，用于生成多样化候选话语，确保估算的稳定性。
影响候选集质量和指标效果。
Open Questions Unanswered questions from this research
- 1 如何进一步优化候选集生成的效率，减少计算成本，提升实时应用能力。
- 2 多模态信息（如语调、手势）对话语预测性的影响机制尚未充分研究。
- 3 不同文化和语言背景下，预期模型的普适性和适应性仍需验证。
Applications
Immediate Applications
对话系统优化
利用信息价值指标提升对话生成的自然性和合理性，使系统更符合人类预期，改善用户体验。
阅读理解模型
结合多维预测性指标，增强模型对文本中关键信息的捕捉能力，提高理解准确率。
Long-term Vision
人机交互的认知模拟
打造更接近人类认知的语言理解系统，实现自然、流畅的交流，推动智能助手发展。
Abstract
We present information value, a measure which quantifies the predictability of an utterance relative to a set of plausible alternatives. We introduce a method to obtain interpretable estimates of information value using neural text generators, and exploit their psychometric predictive power to investigate the dimensions of predictability that drive human comprehension behaviour. Information value is a stronger predictor of utterance acceptability in written and spoken dialogue than aggregates of token-level surprisal and it is complementary to surprisal for predicting eye-tracked reading times.