Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models

Recent advances in large language models, such as GPT-4, T5, and BERT, have demonstrated remarkable capabilities across various NLP tasks. Early efforts focused on short-text understanding, but as models grew larger, the importance of long-text comprehension became evident. Prior studies (e.g., Shaham et al., 2023; Li et al., 2023) evaluated performance on extended inputs but lacked systematic control over variables like input relevance and position. Traditional metrics like perplexity were used to gauge model predictability, yet their correlation with reasoning ability remained unclear. Chain-of-Thought prompting emerged as a promising technique to improve reasoning, but its efficacy diminishes with longer inputs. The gap lies in understanding how input length, independent of task difficulty, impacts reasoning, which this study aims to address.

Despite models supporting longer inputs, their reasoning accuracy often declines sharply beyond certain lengths, typically around 500-1000 tokens. This performance drop hampers practical applications like multi-turn dialogue, long document analysis, and complex knowledge inference. The core challenge is isolating the effect of input size from task complexity and irrelevant information. Existing benchmarks do not adequately control for these factors, making it difficult to attribute performance degradation solely to length. Moreover, traditional evaluation metrics fail to reflect reasoning quality in extended contexts. Addressing this requires a carefully designed dataset and experimental framework that isolates input length as the key variable, enabling precise analysis of its impact on reasoning capabilities.

The study introduces the FLenQA dataset, which systematically varies input length by embedding relevant information within irrelevant background texts at different positions and background types. This multi-task dataset includes three reasoning tasks—MonoRel, PIR, and simplified Ruletaker—each with 100 base instances expanded to lengths of 250, 500, 1000, 2000, and 3000 tokens. The design ensures task difficulty remains constant across lengths, isolating the length effect. Additionally, the study employs multiple background types (duplicate, similar, different) and paragraph placements (start, middle, end, random) to evaluate robustness. Combining this with Chain-of-Thought prompting and traditional next-word prediction metrics, the approach provides a comprehensive view of how input size influences reasoning, failure modes, and model biases.

• �� Construct the FLenQA dataset with three tasks, each with 100 base instances, expanded to multiple lengths (250-3000 tokens). • For each instance, embed key paragraphs within irrelevant background texts, varying background types (duplicate, similar, different). • Disperse key paragraphs at different positions (start, middle, end, random) to simulate diverse input structures. • Generate multiple versions per instance to control for length while keeping task content constant. • Evaluate five models (GPT-4, GPT-3.5, Gemini Pro, Mistral Medium, Mixtral 8x7B) using prompt engineering, including Chain-of-Thought. • Measure accuracy, bias, and failure modes across scenarios, analyzing correlations with traditional metrics. • Conduct ablation studies to understand the impact of background relevance, paragraph placement, and prompt strategies.

The experiments involve evaluating models on the FLenQA dataset across five input lengths, with variations in background relevance and paragraph placement. Metrics include accuracy on reasoning tasks, analysis of failure modes (e.g., ignoring instructions, bias), and correlation with next-word prediction. The evaluation setup maintains consistent prompts, temperature, and sampling methods. Additional ablation studies compare the effects of different background types and paragraph positions. The experimental design ensures that observed performance changes are attributable primarily to input length, providing clear insights into the degradation patterns and failure modes in long-text reasoning. Results are statistically analyzed to confirm significance and robustness.

Models show a consistent decline in reasoning accuracy as input length exceeds 3000 tokens, with average accuracy dropping from 92% to 68%. Chain-of-Thought prompts improve performance in shorter inputs but offer limited mitigation in longer texts, with GPT-4 showing some increased robustness. Traditional next-word prediction accuracy correlates negatively with reasoning performance, indicating that better predictive ability does not equate to stronger reasoning in extended contexts. Models tend to bias responses towards 'False', ignore instructions, or prematurely output answers, revealing fundamental limitations. Background relevance and paragraph placement have minor effects compared to length, confirming length as the dominant factor in performance degradation.

The findings are directly applicable to improving long-text question answering, multi-turn dialogue systems, and knowledge inference tasks. The dataset and evaluation framework can guide training strategies to enhance model robustness in real-world scenarios involving lengthy documents. Industry applications include legal document analysis, scientific literature review, and complex customer service interactions, where understanding and reasoning over extended texts are essential. The insights also inform model architecture design, emphasizing the need for mechanisms that better handle long-range dependencies and reduce bias.

The study focuses on specific tasks and models, limiting generalization. It does not explore training or architectural modifications to mitigate length effects. The dataset simplifies real-world long-text scenarios, which often involve noisy, unstructured data. The internal mechanisms of performance degradation remain unexplored, necessitating future interpretability research. Additionally, computational costs increase with longer inputs, posing practical challenges for deployment. Future work should address these limitations by developing scalable, interpretable, and more realistic long-text reasoning models.