UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
UICoder fine-tunes LLMs for UI code generation using automated feedback, achieving near-proprietary model performance with synthetic data and iterative filtering.
Key Findings
Methodology
This work introduces a method combining automated tools—like SwiftUI compilers and multimodal models—to guide LLMs in generating high-quality UI code. Starting from an existing model, it self-generates a large dataset of UI programs, which is then filtered, scored, and deduplicated using compiler success metrics and CLIP-based relevance scores. The refined dataset is used for iterative fine-tuning, progressively improving the model’s ability to produce compilable, visually relevant code. Multiple iterations enhance performance, with no human annotations required, significantly reducing costs. The approach was applied to open-source models like StarCoder, resulting in models that outperform baseline open models and approach proprietary-level performance.
Key Results
- After five iterations, the models achieved a compilation success rate of 79%, CLIP relevance scores of 0.40, and outperformed all publicly available models in automated metrics and human preference tests. The synthetic dataset comprised nearly one million SwiftUI programs, leading to substantial improvements in code correctness and visual relevance.
- The models maintained high performance across diverse UI descriptions and complex layouts, demonstrating the effectiveness of automated filtering and multimodal scoring. The results show that the approach reduces reliance on human feedback while achieving near state-of-the-art performance.
- Combining compiler success metrics with CLIP relevance scores effectively filters high-quality data, enabling scalable, automated training of UI-generating models, paving the way for practical AI-assisted UI design.
Significance
This research addresses the bottleneck of costly human annotation in training UI code generation models. By leveraging automated tools for data filtering and model fine-tuning, it offers a scalable, cost-effective pathway toward intelligent UI design. The method significantly narrows the gap between open-source models and large proprietary systems, democratizing access to advanced AI-driven UI development. Its impact extends to rapid prototyping, low-code platforms, and automated interface customization, promising to transform software engineering workflows and accelerate digital product innovation.
Technical Contribution
The key technical innovation lies in integrating compiler-based validation and multimodal relevance scoring into an automated, multi-iteration self-training pipeline. This approach enables high-quality synthetic data generation without human labels, leveraging the strengths of code compilation and visual-language models. The use of DPO and top-output ranking for preference alignment further refines the model’s output quality. The method’s modular design allows easy adaptation to other UI frameworks and languages, representing a significant advance in scalable, automated model training for code synthesis.
Novelty
This work is the first to systematically combine automated compilation and multimodal relevance scoring for self-supervised UI code dataset creation and model fine-tuning. Unlike prior approaches relying on human annotations or proprietary distillation, it automates the entire pipeline, achieving performance close to large commercial models. Its iterative, feedback-driven process sets a new standard for scalable, low-cost AI training in UI code synthesis.
Limitations
- The current approach is tailored to SwiftUI and mobile UI platforms; adapting to web or desktop frameworks requires additional dataset and tool support.
- Automated filtering may struggle with highly complex or unconventional layouts, potentially limiting generalization.
- High computational costs for large-scale data generation and iterative training pose barriers for widespread deployment, especially for smaller organizations.
Future Work
Future directions include extending the framework to other UI frameworks like React or Flutter, integrating user feedback for personalized UI generation, and exploring reinforcement learning techniques to further enhance output quality. Improving efficiency and reducing hardware requirements will also be key to broader adoption. Additionally, incorporating real-time user preferences and interactive feedback could enable adaptive, personalized interface design.
AI Executive Summary
Designing user interfaces is a critical yet resource-intensive aspect of software development. Traditional methods rely heavily on manual coding, which is slow, costly, and requires specialized expertise. Recent advances in large language models (LLMs) like GPT-4 and StarCoder have shown promise in automating code generation, but their performance in UI synthesis remains limited due to sparse training data and complex layout understanding.
This paper introduces UICoder, a novel approach that leverages automated feedback—such as SwiftUI compilers and multimodal models—to iteratively improve LLMs for UI code generation. The core idea involves generating a large synthetic dataset of UI programs from descriptions, then filtering, scoring, and deduplicating these samples automatically. The filtered high-quality data is used to fine-tune the model repeatedly, leading to substantial performance gains. This process eliminates the need for costly human annotations, making scalable, high-quality UI code generation feasible.
Experimental results demonstrate that after five iterations, the models achieve a compilation success rate of 79% and a CLIP relevance score of 0.40, outperforming all publicly available models and approaching the performance of large proprietary systems. The models generate complex, visually relevant SwiftUI code from natural language descriptions, supporting diverse UI layouts and interactions. These findings suggest a promising future for AI-assisted UI design, enabling rapid prototyping and low-code development.
The significance of this work lies in its scalable, automated methodology, which addresses the bottleneck of data scarcity and high annotation costs. By integrating compiler validation and multimodal relevance scoring, it sets a new standard for training data quality and model robustness. Future work will focus on expanding to other platforms, improving efficiency, and incorporating user feedback for personalized UI generation. Overall, UICoder paves the way toward more accessible, intelligent interface design tools, transforming how software interfaces are created and deployed.
Deep Dive
Abstract
Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. In this paper, we explore the use of automated feedback (compilers and multi-modal models) to guide LLMs to generate high-quality UI code. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset using an original model, applying automated tools to aggressively filter, score, and de-duplicate the data into a refined higher quality dataset. The original LLM is improved by finetuning on this refined dataset. We applied our approach to several open-source LLMs and compared the resulting performance to baseline models with both automated metrics and human preferences. Our evaluation shows the resulting models outperform all other downloadable baselines and approach the performance of larger proprietary models.