Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments
IGOR combines LLM subtask planning with goal-conditioned asynchronous PPO, outperforming reported IGLU and Crafter baselines.
Key Findings
Methodology
IGOR has a Language Module, Task Manager, and Policy Module. An environment-tuned LLM converts an instruction into ordered subtasks; the Task Manager encodes them, detects completion, assigns rewards, and terminates episodes; an asynchronous PPO policy executes one goal using visual observations and goal codes. Training uses goal-based single-subtask episodes and curriculum sampling.
Key Results
- The method is evaluated in IGLU and Modified Crafter. IGLU contains 109 training and 41 test English instructions and uses F1; the authors report that IGOR exceeds NeurIPS IGLU competition systems, although the supplied text gives no exact F1 score.
- In IGLU, Flan-T5 base predicts coordinate sequences or cuboid primitives, while the policy learns single-block placement or removal. In Crafter, IGOR is reported to surpass Dreamer-v3 baselines, but no numerical score or percentage improvement is provided in the supplied paper text.
- Curriculum sampling changes task probabilities using average reward r_i, variability δ_i, scaling d, and threshold τ: successful tasks receive 1/d, while underperforming tasks receive 1+δ_i·d before softmax normalization. The paper states that curriculum improves final policy quality.
Significance
The work separates language understanding from fine-grained environmental control, addressing two persistent weaknesses: general-purpose LLMs often fail at long-horizon execution and spatial reasoning, while conventional RL is usually tied to narrow task definitions. IGOR offers a practical route to combine pretrained linguistic knowledge with learned embodied behavior under limited supervision. Its relevance extends to game agents, simulation, digital twins, and eventually robotics, where modular replacement and explicit subgoal monitoring are valuable.
Technical Contribution
The principal contribution is an explicit interface between language plans and goal-conditioned control. The Task Manager centralizes subtask encoding, completion detection, reward assignment, and episode termination. PPO receives a POMDP observation augmented with the current goal, while training focuses on one subtask at a time. The paper also compares coordinate and primitive representations in IGLU, introduces textual Crafter instructions, and combines LLM-based augmentation with adaptive curriculum sampling.
Novelty
The novelty is not a new LLM planner or a new PPO variant. It is the environment-oriented decomposition of instruction following into independently trainable planning, task-management, and visual-control modules. Compared with end-to-end agents that consume frozen language embeddings, or systems relying on handcrafted language-to-action translation, IGOR makes subtask supervision, goal conditioning, curriculum learning, and limited-data augmentation first-class design elements.
Limitations
- The supplied manuscript reports qualitative superiority over baselines but does not provide exact IGLU F1 values, Crafter scores, confidence intervals, or significance tests. This prevents a precise estimate of effect size and robustness across random seeds.
- IGLU has only 109 training and 41 test instructions, while Crafter language is generated and stylized with Mistral-7B-Instruct. Such data may contain template bias, limited linguistic coverage, and synthetic artifacts that inflate in-distribution performance.
- Independent module training can create planning-execution mismatch. If the LLM outputs an invalid order or unachievable goal, the Task Manager has no clearly described closed-loop replanning or recovery mechanism.
Future Work
Future work should release complete scores, ablations, confidence intervals, and compute costs; test longer instructions and unseen subtask combinations; and add execution-feedback-driven replanning. Promising directions include hierarchical RL, reward shaping, curiosity-driven exploration, cross-environment transfer, multimodal grounding, and real-robot evaluation with safety constraints and clarification dialogue.
AI Executive Summary
Complex instructions often require an agent to perform many dependent actions. General-purpose LLMs can describe plausible plans but remain unreliable at spatial reasoning and sustained interaction; conventional RL can master fine-grained control but usually lacks open-ended language understanding. The paper introduces IGOR, a hierarchical system designed to connect these complementary abilities.
IGOR contains three modules. The Language Module fine-tunes an LLM to translate an instruction into ordered subtasks. The Task Manager encodes those goals, checks completion, supplies a positive reward of r=+1, and ends or advances the episode. The Policy Module uses asynchronous Proximal Policy Optimization (PPO) to execute one subtask from visual observations. Goal-based training reduces long-horizon complexity, while curriculum sampling emphasizes difficult tasks; ChatGPT augmentation and primitive representations address limited language data.
Experiments cover IGLU and Modified Crafter. IGLU uses 109 training and 41 test instructions for three-dimensional block construction; Crafter requires resource collection, crafting, and combat. The authors report that IGOR outperforms IGLU competition methods based on T5, Pegasus, and BrainAgent, and surpasses Dreamer-v3 Crafter baselines. However, the supplied text does not report exact scores. The main contribution is a modular LLM-to-RL interface; its main risks are small datasets, synthetic-language bias, planning errors, and incomplete quantitative reporting.
Deep Analysis
Background
Prior work has explored LLM prompting, robot planning, language grounding, and text-conditioned RL. Systems such as CLIP-based grounding, Dynalang, Emma, EmBERT, MineDojo, Text-Crafter, and IGLU expand multimodal embodied learning. Yet LLMs lack reliable environmental experience, while end-to-end RL struggles with long instructions and combinatorial state spaces. IGOR targets this interface.
Core Problem
The task is to execute linguistically complex, interdependent instructions in partially observable virtual environments. Bottlenecks include translating language into executable subgoals, resolving spatial relations and ordering, assigning credit across long action sequences, and generalizing from very small instruction datasets. A useful solution must preserve language flexibility without forcing the controller to solve the entire plan at once.
Innovation
- �� Three explicit modules: Language Module, Task Manager, and Policy Module.
- �� LLM output is a subtask plan rather than low-level action commands.
- �� Goal-conditioned PPO learns one objective at a time.
- �� ChatGPT rewriting, primitive grouping, and curriculum sampling mitigate data scarcity and uneven task difficulty.
- �� Modified Crafter adds free-form textual instructions and a corresponding dataset, extending an achievement-based game into an instruction-following benchmark.
Methodology
- �� Input: natural-language instruction plus environment observations.
- �� Planning: Flan-T5 base or an environment-specific LLM maps text to coordinate, color, primitive, or Crafter subtask sequences.
- �� Management: the Task Manager encodes the current goal, detects completion, gives r=+1, and switches or terminates.
- �� Control: asynchronous PPO acts in a POMDP using vision and goal codes; it optimizes E[min(ρ_tÂ_t, clip(ρ_t,1−ε,1+ε)Â_t)].
- �� Curriculum: task sampling uses reward r_i, variability δ_i, scale d, and threshold τ, followed by softmax normalization.
Experiments
IGLU observations include a 64×64×3 point-of-view image, six inventory counts, and pitch/yaw information. Its action space has 13 discrete actions and six continuous movement actions; both walking and flying are tested. The dataset has 109 training and 41 test instructions. Baselines include T5, Pegasus, and BrainAgent, with F1 as the IGLU metric. Crafter maps each original achievement to a subtask and uses Mistral-7B-Instruct for instruction generation and stylization; Dreamer-v3 is the principal comparison.
Results
The authors report that IGOR outperforms competition solutions in IGLU and Dreamer-v3-based baselines in Crafter. In IGLU, coordinate and primitive formats support different planning granularities, while ChatGPT rotation augmentation broadens language-to-structure correspondences. Curriculum learning is reported to improve the final agent. The supplied text does not include exact F1 values, Crafter scores, compute costs, or numerical ablation results, so quantitative ranking cannot be independently reconstructed.
Applications
The design fits game agents, virtual construction, simulation training, digital twins, and robot pretraining. Deployment requires environment-specific subtask definitions, completion detectors, an action interface, and a visual policy. Its modularity allows an organization to replace the LLM or controller independently, but practical systems would also need safety filters, timeout rules, human approval, and recovery from invalid plans.
Limitations & Outlook
The method depends on environment-specific goals and completion predicates, and transfer to new environments is not demonstrated. LLM planning errors can propagate directly to execution, while the policy may overfit known subtask combinations. Generated language may introduce stylistic or semantic artifacts, and asynchronous PPO can be computationally expensive. Stronger systems should add closed-loop replanning, uncertainty estimation, real-world tests, standardized statistical reporting, and robust multimodal grounding.
Plain Language Accessible to non-experts
Think of IGOR as a restaurant handling a complicated order. A customer says, “Defeat two monsters, collect water, and make an iron sword.” The head chef—the language module—does not cook every step. Instead, it rewrites the order into a checklist: defeat one monster, defeat another, collect water, make the sword. A floor manager, the Task Manager, hands one item to the cook and checks whether it is truly finished. The cook, the reinforcement-learning agent, has practiced individual jobs and uses what it sees in the kitchen to act.
Training works like an apprenticeship. Easy jobs come first, while jobs that are still going badly appear more often. This is why a long complicated order can be handled as a sequence of small, checkable jobs rather than one giant guess.
In IGLU, the kitchen becomes a three-dimensional building site where the agent places or removes colored blocks from a description. In Crafter, it gathers resources, crafts tools, and fights creatures. The strength is clear division of labor; the weakness is equally clear: if the head chef misunderstands the order, the cook may execute the wrong plan perfectly.
ELI14 Explained like you're 14
Imagine a sandbox game where a friend messages: “Beat two zombies, get water, then craft an iron sword.” A chatbot may understand the words but still have no idea where to walk or which button to press. IGOR gives the job to two teammates: one understands the mission, and the other actually plays the game.
The first teammate acts like a class monitor. They turn the sentence into a checklist: beat zombie, beat zombie, collect water, craft sword. The second teammate is the gamer. They focus on one checklist item, look at the screen, and act. When the item is done, the monitor gives a reward and moves to the next one. During training, the gamer starts with easy missions and gradually faces harder ones—just like a game tutorial that keeps leveling up.
The researchers tested this in IGLU, where the agent builds block structures from descriptions, and Crafter, where it gathers resources, makes tools, and fights enemies. The paper says IGOR beats several competition systems and Dreamer-v3 baselines, although the supplied text does not show the exact scores.
The cool idea is separating “understand the words” from “control the character.” The danger is that a wrong checklist can lead to very successful wrong behavior. A future version should keep checking the game state and rewrite the plan when something goes off track!
Glossary
Large Language Model (LLM)
A large neural network trained to understand and generate language. Here it converts natural-language instructions into high-level subtasks rather than directly producing motor actions.
The Language Module uses Flan-T5 base and LLM-based augmentation through ChatGPT or Mistral-7B-Instruct.
Goal-Conditioned Reinforcement Learning
A learning setting in which a policy receives both environmental observations and an explicit goal. The policy learns actions that achieve that particular goal.
IGOR trains the controller on one subtask per episode or decision sequence.
Proximal Policy Optimization (PPO)
A policy-gradient algorithm that clips the ratio between new and old action probabilities, limiting overly large updates. This usually improves training stability.
The Policy Module uses asynchronous PPO and the clipped surrogate objective specified in the paper.
Task Manager
The intermediary that connects language planning to environmental control. It encodes goals, detects completion, assigns rewards, advances the plan, and terminates episodes.
It supplies r=+1 when a subtask is successfully completed.
Curriculum Learning
A training strategy that changes the distribution of examples according to agent competence. Difficult or poorly solved tasks are sampled more frequently.
IGOR adjusts subtask probabilities using reward statistics, variability, and threshold τ.
IGLU
A virtual 3D environment in which an embodied agent constructs colored block structures from natural-language descriptions. Performance is measured with F1.
The paper uses 109 training and 41 test instructions.
Open Questions Unanswered questions from this research
- 1 Exact scores, confidence intervals, random-seed variation, and significance tests are absent from the supplied text, so the magnitude and reliability of the reported improvements remain unclear.
- 2 The consequences of independently training the planner and controller—especially error propagation, recovery, and replanning over long horizons—are not fully established.
- 3 It remains unknown whether synthetic Crafter language reflects real users and whether the learned interface transfers to unseen environments or physical robots.
Applications
Immediate Applications
Game and simulation agents
Game developers can define detectable achievements or subtasks, train an LLM planner, and use a PPO controller for resource gathering, construction, and combat. The environment must expose safe actions, reliable completion tests, and episode-failure rules.
Digital-twin operations
In simulated warehouses or factories, users could issue multi-step natural-language orders while the planner converts them into procedures and the policy acts from visual state. Deployment requires state sensors, completion verification, safety filters, and human approval for risky actions.
Long-term Vision
Language-driven robotics
A future IGOR-like system could let people request sorting, transport, or assembly tasks conversationally. Major barriers include noisy perception, physical safety, unknown objects, invalid plans, and sim-to-real transfer; closed-loop feedback and hard constraints will be essential.
Abstract
In this study, we address the issue of enabling an artificial intelligence agent to execute complex language instructions within virtual environments. In our framework, we assume that these instructions involve intricate linguistic structures and multiple interdependent tasks that must be navigated successfully to achieve the desired outcomes. To effectively manage these complexities, we propose a hierarchical framework that combines the deep language comprehension of large language models with the adaptive action-execution capabilities of reinforcement learning agents. The language module (based on LLM) translates the language instruction into a high-level action plan, which is then executed by a pre-trained reinforcement learning agent. We have demonstrated the effectiveness of our approach in two different environments: in IGLU, where agents are instructed to build structures, and in Crafter, where agents perform tasks and interact with objects in the surrounding environment according to language commands.