Looking beyond the next token
TRELAWNEY rearranges training sequences, raising Star Graph accuracy from 0.05 to 1.00 on G(20,5).
Key Findings
Methodology
TRELAWNEY changes data, not the decoder-only Transformer or training stack. At a decision point d, it inserts future tokens z between <T> and </T>. Training mixes original D and augmented Daug as D'=pD+(1-p)Daug, using standard teacher-forced cross-entropy while masking only the loss on <T>. The model therefore learns both future goals and the path leading to them.
Key Results
- On Star Graph, NTP obtains only 0.05 autoregressive accuracy on G(20,5), whereas both TRELAWNEY-fixed and -random reach 1.00. On the longer G(2,10), random with model-generated goals reaches 0.91 versus 0.50 for NTP.
- On CLRS-Text Tarjan strongly-connected-components tasks, NTP scores 0.27 on scc-15. TRELAWNEY-random reaches 0.512 with generated goals and 0.544 with specified goals; rule-based augmentation reaches 0.34 and 0.47.
- On Tiny Stories, models train on 300,000 examples for one epoch. GPT-4 judges 100 stories per model across six trials and prefers TRELAWNEY-implicit-pos over NTP few-shot generations with a reported 76.53% win rate.
Significance
The paper reframes poor planning as a data-factorization problem rather than an unavoidable architectural defect. By exposing future information during training, it targets the Clever Hans shortcut, weak gradients on early indecipherable tokens, and exposure bias. The approach could make planning-capable language models cheaper to build because it preserves standard Transformer infrastructure and decoding. Its broader value will depend on replication in open-domain corpora and real decision-making environments.
Technical Contribution
The contribution is a data-centric sequence-rearrangement framework combining copied future spans, optional natural-language positional metadata ζ(k,z), and a mixture of regular and augmented examples. Masked cross-entropy retains ordinary language-modeling behavior while training prediction of </T>. At inference, users can run standard autoregression, autonomous <T>-generation, or goal-conditioned generation with specified z. Thus future planning becomes an interface rather than a new prediction head or specialized decoder.
Novelty
Unlike multi-token prediction methods such as Gloeckle et al. and architectural bidirectional-prefix/suffix approaches such as Hu et al., TRELAWNEY keeps the model and off-the-shelf decoding unchanged. Its fundamental innovation is to alter the order in which supervision appears, causing difficult early branch decisions to receive future-oriented training signals. The paper’s novelty is therefore a new use of data factorization as an inductive bias for planning.
Limitations
- Evidence is limited to synthetic Star Graphs, a CLRS-Text subset, and Tiny Stories. Decision points, future spans, sentence boundaries, and target templates are task-designed, so generalization to open-domain long-form reasoning remains unestablished.
- The paper does not fully characterize compute overhead, scale, or sensitivity to p, target length, and insertion frequency. Random augmentation outperforming rule-based selection in CLRS-Text is empirically interesting but theoretically unexplained.
- Story quality is assessed mainly through GPT-4 preference judgments, and the supplied paper excerpt truncates the reporting. Human ratings, factuality, perplexity, and diversity need fuller evaluation.
Future Work
Future work should test large-scale pretraining, code, and real agent trajectories; relate model capacity to target-generation ability; and analyze p, span length, and decision-point selection. The authors also motivate agentic behaviors through ζ, autonomous long-term goals, and goal-conditioned planning. Useful extensions include automatic discovery of branching points, goal verification, search, reflection, and theoretical analysis of error propagation.
AI Executive Summary
Causal language models are trained as token-by-token predictors. Teacher forcing makes this process stable and parallelizable, but it also supplies the correct prefix at every step. A model may therefore exploit local clues rather than learn the difficult early decision that determines a long sequence. At inference, the missing plan can produce exposure bias and cascading errors.
Thankaraj, Jiang, Kolter, and Bisk propose TRELAWNEY, a data-rearrangement method. At selected decision points, future text is inserted between <T> and </T>; it may be a copied future span or a natural-language target such as “I want the [k]th sentence from here to be...”. Training mixes ordinary and augmented data and masks only the loss on <T>, preserving standard cross-entropy, parallel training, Transformer architecture, and ordinary decoding.
The results span planning, algorithmic reasoning, and stories. On Star Graph G(20,5), NTP achieves 0.05 accuracy while TRELAWNEY reaches 1.00; on longer G(2,10), random augmentation with model-generated goals reaches 0.91 versus 0.50 for NTP. On CLRS-Text Tarjan strongly-connected-components tasks, scc-15 improves from 0.27 with NTP to 0.512 with generated goals and 0.544 with specified goals. On Tiny Stories, GPT-4 prefers TRELAWNEY generations with a reported 76.53% win rate. The evidence is promising but remains small-scale and task-engineered; open-domain and real-agent validation are essential.
Deep Analysis
Background
Standard autoregressive modeling factorizes Pθ(y)=∏t pθ(yt|y<t). Teacher forcing provides stable gradients and hardware-efficient parallelism, but Bachmann and Nagarajan identify long-range failures and “Clever Hans” shortcuts. Multi-token prediction, including Gloeckle et al., changes prediction granularity; Hu et al. modify architecture and directionality. TRELAWNEY instead asks whether the data order itself can encode planning bias.
Core Problem
Humans often establish a goal before selecting exact wording or intermediate actions, whereas NTP predicts each next token from a ground-truth prefix. This lets later tokens reveal the answer and deprives the earliest branch of learning signal. The resulting indecipherable token may be predicted poorly at inference, where self-generated history replaces the clean training prefix and errors compound.
Innovation
TRELAWNEY transforms y into y1…yd<T>z</T>yd+1…yT. Copying inserts a contiguous future span; ζ(k,z) adds positional or semantic instructions. Fixed uses consistent spans, random samples them per example, and rule-based selection uses task knowledge. The method differs from multi-token prediction and Hu et al.’s architectural changes because it preserves the decoder, objective family, infrastructure, and standard decoding algorithms.
Methodology
- �� Input: sequence y, decision point d, future span z, and optional ζ(k,z).
- �� Augmentation: construct aug(y) by inserting <T>z</T> before the remaining sequence.
- �� Distribution: train on D'=pD+(1-p)Daug to retain conventional language modeling.
- �� Loss: L=-E[|y|⁻¹∑j I(yj≠<T>)logP(yj|y<j)]. The opening marker is masked; </T> remains supervised.
- �� Inference: use standard autoregression, insert <T> and let the model generate z, or provide z externally before continuing.
- �� Task design: exclude v1 and vgoal in Star Graph; use Tarjan trace states in CLRS-Text; use sentence-level goals in Tiny Stories.
Experiments
Star Graph uses 200,000 programmatically generated examples and 5,000 held-out examples per graph, comparing NTP, fixed, random, generated goals, and specified goals. CLRS-Text uses 60,000 training examples from the strongly-connected-components subset and 500 test examples per graph length, comparing random and rule-based spans. Tiny Stories uses 300,000 examples for one epoch; GPT-4 judges 100 anonymized stories per model over six trials with 95% binomial confidence intervals.
Results
For G(2,5), G(5,5), G(20,5), and G(2,10), NTP scores 0.50, 0.20, 0.05, and 0.50. TRELAWNEY-random with generated goals scores 1.00, 1.00, 1.00, and 0.91. On CLRS scc-15, NTP is 0.27, random generated-goal inference is 0.512, and specified-goal inference is 0.544. Larger-model ablations indicate future-token planning improves with capacity.
Applications
Potential uses include code generation with intermediate plans, algorithmic and mathematical reasoning, workflow planning, controllable dialogue, and long-form narrative generation. The main prerequisite is a meaningful decision point and a future target representation. Because training uses ordinary cross-entropy and inference supports off-the-shelf decoding, deployment could fit existing language-model pipelines.
Limitations & Outlook
The evidence comes from synthetic graphs, CLRS-Text, and Tiny Stories rather than broad web-scale or interactive data. Target selection and sentence parsing require task assumptions, while generated goals can themselves be wrong. The work leaves compute scaling, optimal mixture weight p, target length, factuality, and human-quality evaluation underexplored. Future research should test real agents, automatic branch discovery, goal verification, and formal accounts of why random spans regularize learning.
Plain Language Accessible to non-experts
Imagine training a cook. In ordinary practice, the recipe shows every correct previous step and asks the cook to guess only the next cut or stir. The cook can copy the recipe without understanding what the finished dish should be. TRELAWNEY places a small note in the middle: “A few steps from now, make sweet-and-sour soup.” The cook first sees the destination, then learns how to reach it. The original recipes remain in the training pile, so ordinary skills are not discarded. During real cooking, the cook may invent a destination or follow the customer’s requested one. In the experiments, ordinary training succeeded on only 5% of the hardest G(20,5) paths, while TRELAWNEY reached 100%; on longer G(2,10), self-generated intermediate goals reached 91%. The idea is promising, but it still depends on choosing useful notes and has not yet been proven for every kind of recipe.
ELI14 Explained like you're 14
Think of a maze game. A normal language model learns only “which square comes next,” and during practice someone keeps showing it the correct route so far. It may look brilliant, but when playing alone it can panic at the first fork. TRELAWNEY adds a little future clue: “In a few moves, reach the blue room.” The model then practices both the destination and the route.
The clue is wrapped in <T> and </T>. The computer can invent a mini-goal, or you can type one yourself. That is like checking the mission objective before deciding where to move. Instead of blindly guessing the next step, the model gets a chance to organize the path.
The numbers are pretty striking: on G(20,5), normal next-token training succeeds only 5% of the time, while TRELAWNEY reaches 100%. On the longer G(2,10), it reaches 91% when it creates its own clue. For Tarjan’s graph algorithm, the hardest scc-15 score rises from 27% to about 51–54%.
Is it magic? Nope! Researchers still have to decide where to place clues, how long they should be, and whether the model’s invented goal is correct. So the next challenge is testing the trick on coding, real planning, and messy everyday tasks.
Glossary
Next-token prediction
A model predicts the next vocabulary item from previous tokens. Technically, it learns the autoregressive factorization Pθ(y)=∏t pθ(yt|y<t).
It is the paper’s baseline and the objective TRELAWNEY seeks to improve without replacing.
Teacher forcing
Training feeds the ground-truth previous token rather than the model’s own prediction. This stabilizes optimization but creates a train–test mismatch.
The paper links it to Clever Hans behavior, indecipherable tokens, and exposure bias.
TRELAWNEY
A data augmentation and sequence-rearrangement technique that inserts future spans between <T> and </T>. It supplies future-oriented supervision without architectural changes.
It is the central method for planning and controllable generation.
Clever Hans cheat
A model uses answer-bearing ground-truth prefixes as shortcuts instead of learning the underlying plan. The name describes apparently intelligent behavior based on leakage.
Star Graph augmentation excludes the hard first node v1 to reduce this shortcut.
Exposure bias
The distribution mismatch between training on correct histories and inference on self-generated histories. Early mistakes can consequently cascade.
Future-span training is proposed as a way to strengthen early decisions.
CLRS-Text
A text benchmark containing algorithm names, execution traces, and answers. This paper studies Tarjan’s strongly-connected-components subset.
It tests planning across multiple branching algorithmic states.
Open Questions Unanswered questions from this research
- 1 The best future-span length, insertion frequency, mixture weight p, and decision-point policy remain unknown. Random spans sometimes beat rule-based spans, but the regularization mechanism has no established theory.
- 2 It is unclear whether explicit goals improve factuality and reliability in open-domain models or merely create another channel through which incorrect plans can propagate.
Applications
Immediate Applications
Controllable stories and dialogue
A content team can insert sentence-level future goals during fine-tuning, allowing users to request a later plot event or conversational direction. Existing Transformers and decoding software remain usable, but reliable segmentation and goal templates are required.
Code and algorithm planning
A coding assistant can expose intermediate data structures, test outcomes, or graph states as future subgoals. This is especially suitable for programs and algorithms with explicit execution traces, where goals can be checked automatically.
Long-term Vision
Long-horizon autonomous agents
Future systems could propose multistep goals, act toward them, verify progress, and revise plans in a lightweight goal–action–feedback loop. Major obstacles include incorrect self-generated goals, reliable verification, and distribution shift in real environments.
Abstract
The structure of causal language model training assumes that each token can be accurately predicted from the previous context. This contrasts with humans' natural writing and reasoning process, where goals are typically known before the exact argument or phrasings. While this mismatch has been well studied in the literature, the working assumption has been that architectural changes are needed to address this mismatch. We argue that rearranging and processing the training data sequences can allow models to more accurately imitate the true data-generating process, and does not require any other changes to the architecture or training infrastructure. We demonstrate that this technique, Trelawney, and the inference algorithms derived from it allow us to improve performance on several key benchmarks that span planning, algorithmic reasoning, and story generation tasks. Finally, our method naturally enables the generation of long-term goals at no additional cost. We investigate how using the model's goal-generation capability can further improve planning and reasoning. Additionally, we believe Trelawney could potentially open doors to new capabilities beyond the current language modeling paradigm.