MMInA: Benchmarking Multihop Multimodal Internet Agents

The rapid growth of internet content has driven research into autonomous web agents capable of understanding and interacting with complex web environments. Early efforts like WebGPT and WebNav focused on static webpage understanding using reinforcement learning and imitation learning, achieving initial success in simple tasks. However, these approaches often relied on fixed datasets and static environments, limiting their applicability to real-world scenarios where web content evolves continuously. Recent advances in multimodal pretraining, such as VisualBERT and ViLT, have improved understanding of combined visual and textual data, but integrating these into long-range, multi-site reasoning remains challenging. Existing benchmarks like WebShop and Meta-World provide some evaluation platforms but lack the dynamic, multimodal complexity of real web environments. Consequently, there is a pressing need for realistic, comprehensive benchmarks that reflect the true nature of web navigation and information extraction tasks.

The core challenge lies in designing a benchmark that accurately captures the complexity of real-world web environments, including content evolution, multimodal data, and multi-site interactions. Current models perform poorly on multihop tasks, with success rates below 40%, mainly due to early-hop failures and difficulty in maintaining context over long sequences. These limitations hinder progress in developing truly autonomous, intelligent web agents capable of handling practical user tasks like shopping, travel planning, or research. Additionally, existing models lack mechanisms to effectively remember and utilize past actions, leading to fragmented reasoning and reduced success in multi-step scenarios. Addressing these issues requires innovative task design, evaluation protocols, and model architectures that can handle the intricacies of real web content.

This work introduces several key innovations: 1) a benchmark built on real, evolving web data, capturing the dynamic nature of internet content; 2) a comprehensive multi-hop task protocol that assesses long-range reasoning and cross-site information integration; 3) a memory replay mechanism that reintroduces past actions to sustain context, significantly reducing early-hop failures. The framework employs Transformer-based multimodal models, combining textual, visual, and video data, trained with multi-task objectives to enhance reasoning. The evaluation protocol quantifies success through a multi-hop completion metric, providing a nuanced understanding of agent performance. These innovations collectively push the boundaries of current web understanding and reasoning capabilities.

• �� Data collection involves crawling real-world websites across domains like shopping and travel, capturing multimodal content and web evolution over time.
• �� Task design creates 1050 human-authored, multi-step problems requiring multimodal extraction and cross-site reasoning.
• �� Models utilize Transformer architectures (e.g., ViLT, VisualBERT) for multimodal encoding, combined with heuristic web agents for navigation.
• �� A multi-hop success protocol measures the proportion of tasks completed successfully, considering partial progress.
• �� Memory augmentation stores past action trajectories, which are replayed during reasoning to maintain context.
• �� Training employs multi-task learning, optimizing for multimodal understanding, reasoning depth, and action accuracy.
• �� Evaluation includes success rate, information accuracy, and robustness under varying web content dynamics.

Experiments are conducted on a custom dataset of real web pages, with tasks spanning shopping and travel. Baseline models include standard Transformer-based multimodal models and enhanced variants with memory replay. Metrics include success rate, multimodal extraction accuracy, and reasoning depth. Ablation studies compare models with and without memory augmentation, across different task complexities and web content volatility. Results show that memory replay improves success rates by approximately 17%, especially in longer chains. The models are tested under various web update scenarios to assess robustness. Hyperparameters are tuned for optimal multimodal fusion and reasoning depth, ensuring fair comparisons. The experiments validate the effectiveness of the proposed framework in realistic, dynamic environments.

The primary result shows success rate improvements from 35% to 52% in multihop tasks with memory replay. Single-hop accuracy reaches 85%, surpassing baseline models at 78%. Multimodal extraction accuracy improves by 12%, indicating better content understanding. Ablation studies reveal that memory replay reduces early-hop failures by 20%, especially in complex tasks. Performance remains stable across web content updates, demonstrating robustness. These findings confirm that the proposed approach effectively addresses key challenges in real web reasoning, setting new standards for future research.

This benchmark enables the development of more capable autonomous web agents for practical applications such as automated shopping assistants, travel planning bots, and research tools. Industries can leverage these models to improve user engagement, automate routine tasks, and enhance personalized services. Academic research benefits from a standardized platform to evaluate multimodal reasoning and long-term web understanding. Future integration with reinforcement learning and real-time adaptation could further transform web automation, making AI agents more intelligent and versatile in handling complex, evolving online environments.

Despite promising results, current models still face difficulties with highly noisy or rapidly changing web content, limiting real-time deployment. The computational cost of large Transformer models remains high, constraining scalability. The focus on shopping and travel domains restricts generalization; broader domain validation is needed. Additionally, the memory replay mechanism, while effective, introduces latency and complexity. Future work should aim to optimize model efficiency, expand domain coverage, and develop adaptive mechanisms for real-time web content changes, moving closer to practical, scalable AI web assistants.