FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
FigmaTrace enhances VLMs in design tasks using a design phase conversion method.
Key Findings
Methodology
The study employs a design phase conversion method to transform 200 hours of video data into 3469 design trajectories. Four models are trained using the FigmaTrace dataset, demonstrating performance improvements across four GUI environments.
Key Results
- Models trained on FigmaTrace show a 46% improvement on GUI-Odyssey.
- Design phase conversion method outperforms length conversion by 7.3%.
- QWEN3.8-27B model improves by 11.8% on AndroidControl.
Significance
This research fills the performance gap in subjective design tasks for VLMs by providing a high-quality design dataset, advancing automation in the design field.
Technical Contribution
Introduces a design phase conversion method that improves video-to-trajectory conversion accuracy, surpassing existing length conversion methods, offering a new training framework for design tasks.
Novelty
First to apply design phase conversion in VLM training, significantly enhancing model performance in complex design tasks.
Limitations
- Dataset may not cover all design styles, affecting model generalization.
- Conversion method requires high video quality; low resolution may impact results.
Future Work
Future work could expand the dataset to cover more design styles and optimize conversion methods for low-quality videos.
AI Executive Summary
FigmaTrace captures creative nuances in human design workflows, enhancing VLM performance in design tasks. Existing models underperform in subjective design tasks due to a lack of high-quality human workflow data. The FigmaTrace dataset includes 200 hours of video data, converted into 3469 design trajectories using a design phase conversion method. Experiments show significant performance improvements in multiple GUI environments for models trained on FigmaTrace. This research not only provides a new dataset for design tasks but also proposes an effective conversion method, advancing automation in the design field. Nonetheless, the dataset's coverage and conversion method's adaptability need further optimization.
Deep Analysis
Background
Vision Language Models excel in objective tasks but struggle in subjective design tasks. Existing research attempts to improve through human preference alignment and reinforcement learning but lacks high-quality design datasets.
Core Problem
VLMs underperform in subjective design tasks due to a lack of data capturing human design preferences and decisions.
Innovation
Introduces the FigmaTrace dataset, enhancing model performance in design tasks through a design phase conversion method, surpassing existing length conversion methods.
Methodology
- �� Define design skill taxonomy
- �� Collect 200 hours of video data
- �� Convert into 3469 design trajectories
- �� Train four models using FigmaTrace
- �� Conduct performance comparison and ablation studies
Experiments
Experiments use the FigmaTrace dataset to train four models and evaluate performance across four GUI environments. Compare existing length conversion methods and conduct ablation studies.
Results
Models trained on FigmaTrace excel in multiple tasks, notably improving performance on GUI-Odyssey and AndroidControl.
Applications
This research can enhance automation in design tasks, aiding the development of smarter design tools.
Limitations & Outlook
Dataset coverage is limited, potentially affecting model generalization. Conversion method requires high video quality; low resolution may impact results.
Plain Language Accessible to non-experts
Imagine you are cooking in a kitchen. You need to prepare ingredients, cook, and plate according to a recipe. FigmaTrace is like a detailed recipe, helping Vision Language Models learn how to complete complex design tasks. By observing human designers' workflows, the model can better understand how to select appropriate design elements, adjust layouts, and styles, just like a chef creating delicious dishes based on ingredients and spices.
ELI14 Explained like you're 14
Imagine you're playing a design game. You need to choose the right tools and materials according to the task requirements to complete a creative design. FigmaTrace is like a super guide, showing you how to complete the task step by step. By watching professional designers' videos, the model can learn how to choose the right colors, layouts, and elements, just like you choose the right gear and skills in a game. Isn't that cool?
Glossary
Vision Language Model
A model that combines visual and language information for task processing.
Used for automating design tasks.
Design Phase Conversion
A method that converts video data into design trajectories based on design phases.
Improves video-to-trajectory conversion accuracy.
FigmaTrace
A design dataset containing 200 hours of video data.
Used for training Vision Language Models.
Ablation Study
Evaluates model performance changes by removing certain components.
Validates the effectiveness of the design phase conversion method.
GUI Environment
The operational environment of a graphical user interface.
Used to evaluate model navigation capabilities.
Open Questions Unanswered questions from this research
- 1 How to expand the dataset to cover more design styles?
- 2 How to optimize conversion methods for low-quality videos?
Applications
Immediate Applications
Design Task Automation
Helps develop smarter design tools to improve design efficiency.
Long-term Vision
Comprehensive Automation in Design
Advances automation in the design field, enabling more complex design tasks.
Abstract
Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as \textsc{Claude-Opus-5} and \textsc{GPT-5.6-Sol} on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing \textsc{Qwen3.8-27B} outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.