ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

The evolution of robotic learning has seen significant advances in visual perception, navigation, and language understanding, exemplified by platforms like MuJoCo, Habitat, and VQA datasets. These efforts have addressed specific sub-tasks but lack a unified approach for abstract reasoning and cross-modal transfer. TextWorld provides a simplified text-based environment for policy learning, while ALFRED introduces complex visual household tasks with natural language instructions. However, integrating these modalities for scalable, generalizable learning remains a challenge. Prior work often relied on static datasets or end-to-end training, which limits transferability and sample efficiency. This research aims to bridge this gap by establishing a framework where abstract policies learned in text can be effectively transferred to visual environments, enabling more flexible and scalable robot learning.

The core challenge is enabling robots to learn high-level, language-based strategies in a simplified environment and then transfer these strategies to complex, high-dimensional visual scenes. Existing methods struggle with domain gaps, such as differences in perception, action space, and scene variability. This results in poor generalization, especially in unseen environments. The difficulty lies in designing a system that can abstractly reason about tasks, adapt to new layouts, and handle perception errors, all while maintaining efficiency and robustness. Addressing this problem is crucial for developing autonomous agents capable of real-world deployment, where environments are diverse and unpredictable.

This work introduces ALFWorld, a parallel simulation platform that aligns text and visual environments, allowing high-level policy learning and transfer. The key innovations include:

1) A modular, multi-component BUTLER agent that separates language understanding, perception, and control, facilitating independent improvements.

2) Use of PDDL for scene description, enabling flexible environment modeling.

3) Application of imitation learning (DAGGER) to train abstract policies efficiently.

4) Cross-modal transfer demonstrating that strategies learned in TextWorld significantly improve performance in unseen visual scenes.

5) Incorporation of natural language goal descriptions, enhancing language generalization.

These innovations collectively enable scalable, generalizable, and efficient multi-task learning for embodied agents.

• �� Construct ALFWorld by extending TextWorld with PDDL scene descriptions aligned with ALFRED scenes.
• �� Use Transformer-based encoder-decoder models for high-level policy generation, trained via DAGGER with expert demonstrations.
• �� Implement Mask R-CNN detectors to convert visual frames into textual descriptions, providing state estimates.
• �� Develop a low-level controller employing A* path planning and primitive actions (MOVEAHEAD, ROTATELEFT, etc.) to realize high-level commands.
• �� Train the abstract policy in TextWorld, then transfer to ALFRED, leveraging the modular design for incremental improvements.
• �� Conduct multi-task training across diverse scene types, evaluate zero-shot transfer on unseen environments, and compare with baseline models.
• �� Incorporate natural language goal descriptions to test language generalization, and analyze performance across different scene complexities.

Experiments utilize the ALFRED dataset with tasks like pick-and-place, cleaning, heating, and cooling, across 120 rooms with varied object placements. The models are trained in TextWorld with imitation learning, then evaluated zero-shot in ALFRED. Metrics include success rate and goal-condition success rate. Baselines include vision-only models and oracle variants with perfect perception. Additional tests involve natural language goals and different training strategies (pure embodied, pure TextWorld, hybrid). The setup emphasizes cross-modal transfer, generalization to unseen environments, and efficiency, with hyperparameters tuned for optimal performance. Ablation studies assess component contributions and robustness.

Pre-trained BUTLER achieves 26% success in unseen ALFRED environments, outperforming vision-only models by 7x and training 7 times faster. The transfer results confirm that abstract policies learned in TextWorld generalize effectively, especially when combined with natural language goals. Ablation shows that modular design and imitation learning significantly boost performance. Oracle and human goal experiments demonstrate the potential for real-world applicability. Despite detection errors and static map assumptions, the approach markedly improves cross-modal generalization, setting new benchmarks for embodied AI.

This framework can be applied to household robots, warehouse automation, and assistive devices, enabling them to learn tasks abstractly and adapt quickly to new environments. It requires minimal environment-specific tuning, making it suitable for scalable deployment. Long-term, it could lead to autonomous systems capable of lifelong learning, continuously refining strategies through interaction, and operating reliably in dynamic, real-world settings.

Current reliance on Mask R-CNN detection introduces sensitivity to perception errors. Static path planning limits adaptability to dynamic scenes. The simulation-to-reality gap remains significant, requiring further research for real-world transfer. Computational costs are high, especially during training. Future work should focus on perception robustness, environment modeling, and real-world validation to address these challenges.