MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation

With the advancement of AI technologies, the field of web design is continuously evolving. Traditional webpage generation methods primarily rely on code-generation techniques, synthesizing HTML/CSS code by parsing user natural language requests. However, such methods have limitations in handling multimodal content, as webpages are not purely text and code but include heterogeneous elements like images, videos, and charts. These elements' content, style, and geometry must cohere with the global layout and semantic intent. Existing pipelines typically populate these elements via retrieval or placeholders, generating or inserting assets independently, which often leads to style inconsistency and poor global coherence. To address these challenges, MM-WebAgent proposes a novel hierarchical multimodal webpage generation framework that optimizes multimodal content generation and integration through hierarchical planning and self-reflection mechanisms.

The core problem in multimodal webpage generation is how to coordinate the generation of heterogeneous elements to ensure they are visually and semantically consistent with the global layout. Traditional code-generation methods often treat multimodal assets as static or externally provided, lacking the ability to generate novel, semantically aligned, and stylistically coherent multimodal content. Additionally, existing webpage generation methods struggle with complex webpage layouts, leading to style inconsistency and poor global coherence. These issues not only affect the visual appeal of webpages but also impact user experience. Thus, achieving coordinated generation and integration of elements in multimodal webpage generation is a pressing challenge.

The core innovations of MM-WebAgent lie in its combination of a hierarchical agentic framework and self-reflection mechanism. Firstly, the method views multimodal generation as a hierarchical planning and optimization process, distinct from traditional code-generation methods. Through global layout planning and local element planning, it ensures that multimodal components are natively integrated into the page structure. Secondly, it introduces a multi-level self-reflection mechanism capable of iteratively optimizing content and layout at both local and global levels. This mechanism performs targeted edits by analyzing the generated webpage screenshots and HTML code, ensuring consistent layout, spacing, and visual style. Additionally, the framework proposes a new benchmark and evaluation protocol for multimodal webpage generation, providing tools for systematic performance assessment.

The hierarchical agentic framework of MM-WebAgent includes the following key steps:

• �� Global Layout Planning: Defines the section hierarchy, spatial organization, and page-level style attributes, introducing explicit placeholders for multimodal components.

• �� Local Element Planning: Constructs local plans based on the global context to guide content generation. Each local plan includes context information and meta attributes, describing the functional role and style guidance of elements.

• �� Plan Execution: Converts the global layout plan into the HTML/CSS structure of the webpage and generates corresponding assets using designated generation tools.

• �� Self-Reflection Mechanism: Optimizes the generated webpage through local, context, and global levels of self-reflection, ensuring each element is semantically correct and visually sound.

The experimental design includes performance evaluation on the MM-WebAgent-Bench and WebGen-Bench benchmarks. MM-WebAgent-Bench tests the diversity and quality of multimodal webpage generation, including informational, analytical, creative, and commercial use cases. WebGen-Bench primarily tests functional backend code, logic, and component completeness. Baseline methods include code-generation and agent-based methods such as OpenAI-GPT-5.1 and Qwen2.5-Coder. Evaluation metrics include global layout, style coherence, aesthetics, and the quality and integration of images, videos, and charts. Ablation studies are conducted to verify the effectiveness of hierarchical planning and self-reflection mechanisms.

Experimental results demonstrate that MM-WebAgent outperforms baseline methods in multimodal element generation and integration. In the MM-WebAgent-Bench benchmark, it achieved an average score of 0.75, significantly higher than code-generation and agent-based baselines, particularly excelling in the generation and integration of multimodal elements such as images, videos, and charts. In the WebGen-Bench test, despite the task primarily testing functional backend code, MM-WebAgent still demonstrated competitive performance, showcasing its capability in complex multimodal generation tasks. Ablation studies indicate that the introduction of hierarchical planning and self-reflection mechanisms significantly improves the coordination of multimodal content and overall performance, particularly in local metrics such as images and videos.

MM-WebAgent has potential value in multiple application scenarios. Firstly, in web design, the method can automatically generate visually consistent and semantically coherent webpages, reducing the workload of manual design. Secondly, in e-commerce platforms, it can generate personalized product display pages, enhancing the user shopping experience. Additionally, in education and training, the method can be used to generate interactive learning materials, improving learning outcomes. To achieve these applications, it is essential to ensure that the generated content is visually and semantically aligned with user needs and provides a good user experience.

Despite its outstanding performance in multimodal webpage generation, MM-WebAgent has some limitations. Firstly, the method may encounter performance bottlenecks when handling extremely complex webpage layouts, especially in scenarios requiring a large number of multimodal elements. Secondly, although the self-reflection mechanism improves content quality, it may increase computational overhead in some cases, affecting generation efficiency. Additionally, the method may require further optimization when handling specific types of multimodal content, such as dynamic videos, to improve generation smoothness and consistency. Future research directions include optimizing the self-reflection mechanism to reduce computational overhead, extending the framework to support more types of multimodal content, and validating it on larger-scale real-world datasets.