CrafText Benchmark: Advancing Instruction Following in Complex Multimodal Open-Ended World

The evolution of multimodal AI has seen significant progress with models like CLIP, FILM, and vision-language navigation systems, enabling better visual-textual integration. Early benchmarks such as BabyAI and CraftAssist focused on static environments with limited linguistic variation, which simplified instruction understanding but restricted real-world applicability. Recent efforts introduced environment dynamics, yet many still rely on templated instructions and limited object interactions, reducing linguistic richness and environmental complexity. To address these limitations, researchers emphasize the need for benchmarks that incorporate diverse language, environment volatility, and complex interactions, aligning AI capabilities closer to human-like understanding and adaptability in real-world scenarios.

Despite advances, current instruction-following systems struggle with environment volatility and linguistic diversity. Static benchmarks do not simulate real-world unpredictability, leading to overfitting and poor generalization. The core challenge is designing environments that support complex, multi-goal tasks with dynamic object interactions and natural language variations, enabling models to learn flexible, robust behaviors. Addressing this gap is crucial for deploying AI in real-world applications like robotics, autonomous vehicles, and virtual assistants, where unpredictability and linguistic complexity are inherent.

The key innovations include: 1) constructing a large, diverse instruction dataset generated via GPT-4, capturing linguistic richness; 2) designing an environment with dynamic object properties and evolving goals, reflecting real-world volatility; 3) integrating multi-task validation functions for precise success measurement; 4) employing JAX-accelerated RL algorithms for efficient training; 5) establishing a dual evaluation protocol for assessing generalization to both linguistic variations and unseen goals. These innovations collectively enable the development of more adaptable, linguistically flexible AI agents capable of operating in complex, changing environments.

• �� Data generation: Define goal templates, expand with GPT-4 for natural language instructions and paraphrases, and create validation functions for each goal.
• �� Environment design: Extend Craftax to include environment dynamics, object property changes, and multi-goal interactions.
• �� Training: Use JAX-based PPO, FiLM, and Dynalang algorithms, leveraging GPU acceleration for large-scale training.
• �� Evaluation: Implement success rate metrics on paraphrased instructions and new goal configurations, with multiple random seeds for robustness.
• �� Analysis: Compare models on success rates, generalization, and environment adaptability, conducting ablation studies to assess components' contributions.

Experiments involve training models on the medium difficulty subset of the dataset, which includes 277 goals and approximately 5-6 instructions per goal. The models are evaluated on test sets with paraphrased instructions and unseen object configurations. Success rates are averaged over 50 seeds to ensure statistical significance. Hyperparameters such as learning rate, batch size, and environment variability are tuned for optimal performance. Results show that models like PPO-T+ outperform baselines, demonstrating the effectiveness of instruction diversity and environment modeling. Ablation studies confirm the importance of environment dynamics and instruction complexity in achieving robust generalization.

PPO-T+ achieved a success rate of 45% on paraphrased instructions and 28% on new object goals, outperforming baselines like Dynalang (15%). The success rate in complex tasks was lower but improved with diverse training. The models demonstrated high linguistic robustness, handling over 3,900 instruction variants. These results confirm that integrating environment dynamics and large instruction datasets significantly enhances the agent’s ability to generalize and adapt, marking a step forward in real-world AI applications.

This benchmark supports development of autonomous robots, virtual assistants, and interactive systems that require understanding complex instructions in changing environments. It enables researchers to evaluate and improve models’ robustness, scalability, and generalization, accelerating deployment in real-world scenarios such as household robotics, industrial automation, and autonomous navigation. Long-term, it aims to facilitate AI systems capable of lifelong learning and adaptation in unpredictable settings.

Despite progress, success rates in highly dynamic or unseen environments remain below 50%, indicating the need for better generalization techniques. The reliance on predefined validation functions limits reasoning depth. High computational costs for training large models pose practical deployment challenges. Future work should focus on integrating reasoning modules, reducing training costs, and expanding task complexity to include multi-step reasoning and long-horizon planning.