Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

The evolution of web interaction reinforcement learning has transitioned from rule-based systems and imitation learning to deep RL approaches. Early works like program synthesis and behavioral cloning provided initial solutions but struggled with scalability and generalization. Shi et al. (2017) introduced MiniWoB, a benchmark for web tasks, highlighting the challenge of sparse rewards and high-dimensional state spaces. Recent efforts integrated neural architectures but still faced sample inefficiency and overfitting. The need for more scalable, data-efficient methods remains pressing, especially for real-world applications like automated form filling and web navigation. This paper builds on these foundations, proposing a novel combination of demonstration-based constraints and deep relational modeling to address these issues.

The core challenge in web RL is the sparse reward problem compounded by vast, unstructured action spaces and complex spatial relationships within web pages. Naive exploration often fails to discover rewarding trajectories, leading to prohibitively high sample costs. Behavioral cloning mitigates this but overfits and lacks generalization, especially in diverse layouts and natural language inputs. Existing methods struggle to balance exploration efficiency and policy flexibility. The key bottleneck is designing a mechanism that constrains exploration without sacrificing adaptability, enabling agents to learn effectively from limited demonstrations in noisy, high-dimensional environments.

The paper introduces three main innovations: (1) environment-blind workflow policies πw, learned via REINFORCE from demonstration sequences, which impose high-level action constraints without environment state dependence; (2) the DOMNET architecture, embedding HTML elements with multi-layer attention to model spatial and hierarchical relations explicitly; (3) a demonstration-driven training pipeline that combines constrained exploration with deep neural policies, drastically reducing sample complexity. These innovations collectively enable efficient learning in web environments, overcoming traditional RL limitations in sparse reward and high-dimensional settings.

• �� From expert demonstrations, extract action sequences and induce a lattice of possible workflows, representing high-level action constraints. • Train a workflow policy πw using REINFORCE, optimizing the likelihood of selecting demonstration-consistent workflows based on reward signals. • During exploration, sample a workflow z from πw, then select actions uniformly from the constrained set zt(st). • Successful episodes are stored in a replay buffer for training a neural policy πn, which employs DOMNET to embed and attend over HTML structures. • DOMNET embeds each DOM element via attributes, spatial neighbors, and tree relations, applying multi-layer attention to capture spatial and hierarchical dependencies. • Train πn with on-policy and off-policy (replay buffer) updates using A2C, enabling efficient policy learning with minimal demonstrations.

Experiments conducted on MiniWoB, MiniWoB++, and Alaska Airlines web tasks involved training with as few as 3-10 demonstrations. Success rates were measured on test episodes, with WGE outperforming pure RL and behavioral cloning by significant margins. Ablation studies confirmed the importance of workflow constraints in reducing overfitting and improving exploration. The models demonstrated robustness to noisy environments, longer horizons, and natural language goals. Hyperparameters such as learning rate, discount factor, and demonstration count were tuned to optimize performance. Results consistently showed that WGE achieved over 90% success with minimal data, validating its sample efficiency and scalability.

WGE combined with DOMNET achieved success rates exceeding 90% across multiple web tasks, outperforming baseline methods by over 30%. In the Alaska Airlines booking scenario, only one demonstration sufficed to reach an average reward of 0.97, far surpassing previous methods requiring dozens of demonstrations. On MiniWoB++, the approach effectively handled longer horizons and noisy inputs, with success improvements of 40-50%. The ablation analysis highlighted the critical role of demonstration-guided constraints in accelerating learning and avoiding overfitting, especially under sparse rewards. Overall, the results demonstrate that the proposed framework drastically reduces training data needs while maintaining high performance.

This approach is immediately applicable to automating web-based tasks such as form filling, email management, and online shopping. It requires minimal demonstrations and can adapt to diverse webpage layouts, making it suitable for deploying AI assistants and automation tools. In the long term, the methodology could extend to autonomous agents in robotics, customer service, and complex decision-making environments, where sparse rewards and high-dimensional states are prevalent. Its ability to learn efficiently from limited supervision makes it a promising foundation for scalable, real-world AI systems.

The workflow policy’s environment-blind nature limits responsiveness to layout changes or dynamic content, potentially reducing effectiveness in highly variable scenarios. Dependence on demonstration quality means poor or limited demonstrations can impair exploration. Scaling to highly complex or real-time web environments remains challenging, necessitating integration with environment perception and natural language understanding. Future work should address these limitations to enhance robustness and generalization.