Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
Designer-RSI evolves procedural memory from user traffic, boosting GenEval2 execution success to 99.3%.
Key Findings
Methodology
The study proposes a continual adaptation framework using a frozen frontier model to operate professional design software, accumulating and refining natural language skills through external procedural memory. The memory expands by acquiring procedures for uncovered subtasks and deepens by revising existing procedures.
Key Results
- GenEval2 execution success on Claude-Sonnet-4 increased from 72.7% to 99.3%, expanding the skill bank from 76 to 139 skills.
- EVOLVE agent achieved win rates of 61.8% to 67.6% across four specialized design benchmarks.
- Skill widening or deepening alone achieved win rates around 49%, while their combination reached 58.5%.
Significance
The study offers a practical route for continual adaptation of agents under noisy, unverifiable feedback, addressing the complexity of long-horizon tasks in professional graphic design.
Technical Contribution
Through the evolution of procedural memory, the study demonstrates how to improve agent execution success without model weight updates, introducing a matched replay gate mechanism to ensure improvements.
Novelty
This is the first application of procedural memory evolution in graphic design, continuously optimizing the skill bank through user traffic, addressing the lack of a reliable success oracle in design tasks.
Limitations
- Procedural memory mechanisms may fail in certain complex design tasks, leading to ineffective skill bank expansion.
- The replay gate mechanism may not completely prevent regression in some scenarios.
Future Work
Future research could explore the application of procedural memory in other domains, such as video editing or music creation, and optimize the mechanisms for skill bank expansion and deepening.
AI Executive Summary
Professional graphic design is a complex task involving multiple interdependent operations. Traditional methods struggle to provide a reliable programmatic oracle to ensure design success. This study proposes a continual adaptation framework using a frozen frontier model to operate design software and accumulate and refine skills through procedural memory. After five rounds of evolution, the skill bank expanded from 76 to 139 skills, with GenEval2 execution success rising from 72.7% to 99.3%. The method performed excellently across multiple design benchmarks, significantly improving win rates. Despite this, procedural memory mechanisms may fail in certain complex tasks, and future research could explore its application in other domains.
Deep Analysis
Background
Recent advances in generative models have significantly improved image synthesis, but professional graphic design requires structured, editable outputs. Existing methods struggle with the complexity of design tasks, especially without a reliable success oracle.
Core Problem
Professional graphic design involves multiple interdependent operations, with outcomes hard to verify. Design tasks lack a reliable success oracle, making supervised learning costly.
Innovation
The study introduces a procedural memory evolution mechanism, continuously optimizing the skill bank through user traffic, addressing the lack of a reliable success oracle in design tasks.
Methodology
- �� Frozen frontier model operates design software
- �� External procedural memory accumulates natural language skills
- �� Matched replay gate mechanism ensures skill improvement
- �� Skill bank expansion and deepening
Experiments
The experimental design includes five rounds of evolution using 1,406 user briefs and 1,869 automatically graded trajectories, evaluating the effects of skill bank expansion and deepening.
Results
GenEval2 execution success increased from 72.7% to 99.3%, with the skill bank expanding from 76 to 139 skills. EVOLVE agent significantly improved win rates across multiple design benchmarks.
Applications
Procedural memory mechanisms can be applied to other creative domains, such as video editing or music creation, enhancing agents' continual adaptation capabilities.
Limitations & Outlook
Procedural memory mechanisms may fail in certain complex design tasks, and the replay gate mechanism may not completely prevent regression. Future research should optimize skill bank expansion and deepening mechanisms.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef needs to prepare a complex dish. Traditional methods are like a recipe, telling the chef what to do at each step, but if the ingredients are insufficient or the steps unclear, the dish may fail. Procedural memory mechanisms are like an experienced chef who can adjust steps based on past experiences to ensure the dish succeeds. Through continuous trial and optimization, the chef's skill bank expands, eventually handling various complex dishes.
ELI14 Explained like you're 14
Imagine you're playing a complex game where you need to complete many tasks to win. Traditional methods are like a guide, telling you what to do at each step, but if the guide isn't detailed enough, you might fail. Procedural memory mechanisms are like an experienced player who can adjust strategies based on past experiences to ensure you win. Through continuous trial and optimization, the player's skill bank expands, eventually handling various complex tasks.
Glossary
Procedural Memory
A mechanism for accumulating and optimizing skills through experience, continuously expanding and deepening the skill bank.
Used in the study to optimize the skill bank of graphic design agents.
Frozen Frontier Model
A model that operates design software without updating weights.
Used in the study to operate over 230 design tools.
Matched Replay Gate
A mechanism to ensure skill improvements without regression.
Used to evaluate the effects of skill bank expansion and deepening.
GenEval2
A benchmark for evaluating image generation quality.
Used to evaluate the execution success of the EVOLVE agent.
Claude-Sonnet-4
A foundation model used for graphic design tasks.
Used in the study to evaluate the performance of the EVOLVE agent.
Open Questions Unanswered questions from this research
- 1 The effectiveness of procedural memory mechanisms in other domains remains unclear and requires further research.
- 2 The replay gate mechanism may not completely prevent regression in some scenarios.
Applications
Immediate Applications
Graphic Design Optimization
Designers can use procedural memory mechanisms to optimize design processes, improving efficiency and success rates.
Long-term Vision
Cross-Domain Applications
Procedural memory mechanisms can be applied to other creative domains, such as video editing or music creation, enhancing agents' continual adaptation capabilities.
Abstract
Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.