See, Think, Act: Online Shopper Behavior Simulation with VLM Agents
Integrating VLM with webpage screenshots and HTML improves online shopper behavior prediction accuracy by over 6%, enhancing realism in multi-modal models.
Key Findings
Methodology
This work introduces a vision-language model (VLM) framework combining HTML textual data and webpage screenshots for online shopping behavior simulation. The model inputs include user action histories, pruned HTML structures, and aligned webpage images. Using the OPeRA dataset, data preprocessing involves HTML element pruning and screenshot alignment to ensure modality consistency. The training employs supervised fine-tuning (SFT) to maximize likelihood of annotated action-rationale pairs, and reinforcement learning (RL) with a hierarchical, difficulty-aware reward structure to refine behavior sequences. The model outputs structured JSON containing predicted actions and rationales, enabling interpretability. The approach leverages the joint encoding capabilities of VLMs like Qwen-2.5-VL-3B-Instruct, integrating multi-modal signals for improved decision-making in complex web environments.
Key Results
- The multi-modal model achieves an exact match accuracy of 44.57% under SFT+RL, surpassing text-only models by over 20%, demonstrating the significant contribution of visual grounding. Action type F1 scores exceed 57%, indicating high classification accuracy for user actions. Ablation studies confirm that webpage screenshots notably improve recognition of fine-grained behaviors such as button labels and input contents. The model's robustness across different user sessions and complex webpage layouts underscores the effectiveness of multi-modal fusion in capturing nuanced human decision processes.
Significance
This research advances the field of human behavior simulation by integrating visual perception into language models, addressing the limitations of prior text-only approaches. The ability to understand webpage layouts, images, and interface elements leads to more realistic and detailed behavior predictions, crucial for applications like personalized recommendation, virtual assistants, and UX evaluation. The technical innovation lies in the seamless fusion of visual and textual data within a unified model, enabling a deeper cognitive mimicry of human web interactions. This work paves the way for more sophisticated AI agents capable of understanding complex visual environments, bridging the gap between artificial and human decision-making in digital spaces.
Technical Contribution
The core technical contribution is the development of a multi-modal behavior prediction framework that combines VLMs with hierarchical reward mechanisms. The model encodes both HTML and webpage screenshots via joint multimodal transformers, enabling the understanding of spatial layout and visual cues. The training strategy integrates SFT for explicit behavior imitation and RL with a difficulty-aware reward scaling (DARS), which emphasizes challenging decision points. Data preprocessing involves HTML pruning and screenshot alignment to ensure modality consistency. The reward design incorporates structured signals for action accuracy and rationale confidence, leading to improved sequence fidelity. These innovations collectively enable the model to generate more human-like, contextually appropriate behaviors in complex web environments.
Novelty
This is the first comprehensive attempt to embed webpage visual information directly into behavior simulation models using VLMs, moving beyond prior text-only approaches. The integration of aligned webpage screenshots with HTML structures within a unified transformer-based framework represents a novel methodology. Additionally, the hierarchical, difficulty-sensitive reward mechanism tailored for web interaction tasks introduces a new dimension in reinforcement learning for behavior modeling. The combination of these elements results in a system that more faithfully mimics human visual attention and decision-making, setting a new standard in online behavior simulation research.
Limitations
- The model's reliance on webpage screenshots may limit performance in scenarios with severe visual clutter or occlusion, reducing generalization in highly complex or dynamic environments.
- The dataset size, though sufficient for initial training, is limited in diversity, which may restrict the model's ability to generalize across different user behaviors and webpage designs.
- Handling very long interaction sequences remains challenging due to information bottlenecks; future work should explore better long-term context encoding strategies.
Future Work
Future directions include enhancing multi-modal fusion techniques, such as incorporating temporal dynamics and user personalization features. Expanding dataset scale and diversity will improve robustness. Additionally, optimizing inference efficiency and interpretability will facilitate deployment in real-world systems. Exploring transfer learning across different web domains and integrating user-specific preferences could further personalize behavior simulation, making AI agents more adaptable and human-like in complex, real-time scenarios.
AI Executive Summary
In the rapidly evolving landscape of e-commerce, understanding and predicting user behavior on web interfaces is crucial for enhancing personalization and interface design. Traditional models primarily rely on textual data such as HTML content and action logs, but these overlook the critical role of visual cues like webpage layout, product images, and interface elements. Recognizing this gap, recent advances in vision-language models (VLMs) offer a promising avenue to incorporate visual perception into behavior prediction.
This paper presents a novel framework that integrates VLMs with web browsing data, specifically webpage screenshots, to simulate online shopper behavior more faithfully. Using the publicly available OPeRA dataset, the authors preprocess the data by pruning HTML structures and aligning webpage images, creating a multi-modal input space. The model employs supervised fine-tuning (SFT) to learn from annotated action-rationale pairs and reinforcement learning (RL) with a hierarchical, difficulty-aware reward system to refine decision-making. The combined approach significantly improves prediction accuracy, achieving over 44% exact match—more than 20% higher than text-only baselines.
Experimental results demonstrate that visual grounding enhances the model’s ability to recognize fine-grained behaviors, such as identifying specific buttons and input fields, which are vital in complex web environments. The multi-modal model also shows robustness across different user sessions and webpage layouts, indicating strong generalization capabilities. These findings underscore the importance of visual context in human decision processes, offering a more realistic simulation of online shopping behaviors.
The significance of this work lies in its potential to revolutionize user modeling, recommendation systems, and interface testing by providing more cognitively aligned AI agents. Its technical innovations include the joint encoding of HTML and webpage images, and a reward mechanism that emphasizes challenging decision points. However, limitations such as dependency on visual clarity and dataset diversity suggest avenues for future research. Moving forward, expanding datasets, improving long-term context handling, and personalizing models will further bridge the gap between artificial and human behaviors in digital environments, pushing the frontier of multimodal AI.
Deep Dive
Plain Language Accessible to non-experts
想象你在一个厨房里做饭。你不仅要看食谱（文字说明），还要观察厨房里的食材、锅碗瓢盆的位置（图片和布局），才能决定下一步做什么。以前的厨师只看食谱，容易出错，现在有了一个聪明的助手，它能同时看食谱和厨房的图片，帮你更快找到食材，做出更好吃的菜。这就像这篇论文，把网页的图片和文字结合起来，让电脑更像人一样理解网页，从而更准确地模拟用户在网页上的操作。它不仅能看懂网页布局，还能理解按钮、图片的作用，像一个真正懂网页的“厨师”。
ELI14 Explained like you're 14
想象你在玩一个网页游戏，你不仅要看文字说明，还要注意图片和界面布局，才能知道下一步怎么做。以前的电脑程序只看文字，不能理解图片，所以做得不太像人。现在，这个新方法就像给电脑装了眼睛和大脑，让它同时看文字和图片，学习人类怎么在网页上操作。比如，点击按钮、滚动页面、看商品图片，都能更像真实用户。实验发现，这样的电脑能比只看文字的更准确预测用户的行为，准确率提高了6%以上。未来，这样的技术可以帮助电商平台更好地理解用户，提供更贴心的推荐和服务。它就像让电脑变得更聪明、更懂网页一样！
Abstract
LLMs have recently demonstrated strong potential in simulating online shopper behavior. Prior work has improved action prediction by applying SFT on action traces with LLM-generated rationales, and by leveraging RL to further enhance reasoning capabilities. Despite these advances, current approaches rely on text-based inputs and overlook the essential role of visual perception in shaping human decision-making during web GUI interactions. In this paper, we investigate the integration of visual information, specifically webpage screenshots, into behavior simulation via VLMs, leveraging OPeRA dataset. By grounding agent decision-making in both textual and visual modalities, we aim to narrow the gap between synthetic agents and real-world users, thereby enabling more cognitively aligned simulations of online shopping behavior. Specifically, we employ SFT for joint action prediction and rationale generation, conditioning on the full interaction context, which comprises action history, past HTML observations, and the current webpage screenshot. To further enhance reasoning capabilities, we integrate RL with a hierarchical reward structure, scaled by a difficulty-aware factor that prioritizes challenging decision points. Empirically, our studies show that incorporating visual grounding yields substantial gains: the combination of text and image inputs improves exact match accuracy by more than 6% over text-only inputs. These results indicate that multi-modal grounding not only boosts predictive accuracy but also enhances simulation fidelity in visually complex environments, which captures nuances of human attention and decision-making that text-only agents often miss. Finally, we revisit the design space of behavior simulation frameworks, identify key methodological limitations, and propose future research directions toward building efficient and effective human behavior simulators.