A Close Look At World Model Recovery In Supervised Fine-Tuned LLM Planners

The evolution of large language models (LLMs) such as GPT, BERT, and Llama has revolutionized NLP tasks, demonstrating remarkable capabilities in understanding, reasoning, and generation. Early efforts focused on language understanding, but recent studies explore their potential in complex reasoning and environment modeling. Works like Li et al. (2021) and Kadlčík et al. (2025) show that LLMs can learn internal representations of meaning and numerical concepts. In planning, traditional algorithms like Fast Downward and PDDL-based methods dominate, but integrating LLMs offers flexible, end-to-end solutions. Prior research indicates that fine-tuning enhances reasoning, yet understanding whether models internalize environment models remains limited. This gap motivates the current investigation into internal representations and their relation to output performance.

The core challenge is determining whether fine-tuned LLMs truly learn environment models—specifically, whether they encode state predicates and action validity internally, and how these internal representations relate to their generative outputs. Existing evaluation methods focus on output accuracy, which may not reflect internal knowledge. The difficulty lies in disentangling internal representations from output signals, especially under different training data distributions. Moreover, understanding how environment coverage during training influences internal knowledge is critical. Addressing these issues is essential for developing transparent, trustworthy AI planning systems capable of generalizing beyond training distributions.

This study introduces a dual evaluation approach: linear probing of hidden states to quantify internal representations, and probabilistic classification based on generated token probabilities. It innovates by applying these methods to models trained on different data distributions—optimal plans, random walks, and enhanced random walks with broader state coverage—and incorporating State-CoT sequences to improve intermediate state understanding. The combination of internal probing and output analysis provides a comprehensive picture of world model recovery, revealing that internal representations can be robust even when output signals are unreliable. This approach advances interpretability in large models, bridging internal knowledge and observable behavior.

• �� Define planning environments (Blocksworld, Logistics) using PDDL, generate datasets with optimal plans and random walks.
• �� Fine-tune gemma2-9b-instruct models on these datasets, with and without State-CoT augmentation.
• �� During training, incorporate chain-of-thought sequences to improve intermediate state tracking.
• �� For evaluation, prompt models on test instances, generate complete plans, and extract hidden states at intermediate steps.
• �� Randomly select plan steps, generate all possible next actions, and prompt models with partial plans plus each candidate action.
• �� Extract activation vectors from each layer for action tokens, and compute token log probabilities.
• �� Train logistic regression probes to predict action validity and state predicate truth values from hidden states.
• �� Use generated token probabilities to classify actions as valid or invalid, comparing rankings to ground truth.
• �� Analyze the influence of data distribution and state coverage on internal representations and output performance.

Experiments involve training models on datasets generated via optimal planning and random walk strategies, with additional enhanced random walks to increase state coverage. Evaluation metrics include F1 scores of linear probes for action validity and predicate prediction, and classification accuracy based on token probabilities. The models are tested on in-distribution and out-of-distribution instances, with ablation studies comparing different data distributions and the effect of State-CoT augmentation. Results are analyzed to assess internal representation quality, correlation with output performance, and the impact of environment coverage. Cross-environment generalization is also examined to evaluate robustness.

Models trained on valid action sequences achieved over 85% F1 in linear encoding of action validity, with some predicates like 'on' reaching 80%. Internal representations remained effective even when output probabilities poorly distinguished valid/invalid actions, indicating knowledge storage beyond output signals. Broader state coverage via random walks improved out-of-distribution performance by over 15%, demonstrating better environment understanding. State-CoT sequences further enhanced intermediate state comprehension, leading to more accurate world model recovery and generalization. These findings confirm that internal representations are crucial for robust planning and environment modeling in LLMs.

This framework can be applied to autonomous robots, intelligent agents, and decision-support systems requiring transparent environment understanding. It enables developers to diagnose internal knowledge and improve model interpretability, essential for safety-critical applications like autonomous driving or healthcare. Additionally, the insights can guide training strategies to enhance environment coverage and internal knowledge storage, fostering more reliable AI systems capable of complex reasoning in real-world scenarios.

The current evaluation is limited to structured environments like Blocksworld and Logistics; applicability to unstructured, real-world domains remains untested. Linear probes may not capture complex, nonlinear internal representations, potentially underestimating the depth of stored knowledge. The models' performance in highly sparse or high-dimensional environments is uncertain, requiring further investigation. Computational costs of probing and training multiple classifiers also pose practical challenges. Future work should explore more sophisticated interpretability tools and real-world applications to address these limitations.