GEMS: Agent-Native Multimodal Generation with Memory and Skills
GEMS framework enhances multimodal generation with memory and skills, enabling Z-Image-Turbo to surpass Nano Banana 2 on GenEval2.
Key Findings
Methodology
GEMS framework consists of Agent Loop, Agent Memory, and Agent Skill. Agent Loop improves generation quality through closed-loop optimization; Agent Memory provides persistent trajectory-level memory, reducing redundancy; Agent Skill offers an extensible collection of domain-specific expertise.
Key Results
- GEMS enables the 6B model Z-Image-Turbo to surpass Nano Banana 2 on GenEval2, with an average performance gain of 14.22 points.
- In downstream tasks, GEMS achieves an average improvement of 14.03 points, significantly outperforming other inference-time scaling methods.
- Ablation studies show that Agent Loop, Memory, and Skill each contribute significantly to performance improvements.
Significance
The GEMS framework, through agentic reasoning and domain-specific expertise, overcomes inherent limitations of foundational models, excelling in complex instructions and downstream tasks. This offers new perspectives and tools for the multimodal generation field.
Technical Contribution
GEMS introduces persistent Agent Memory and extensible Agent Skill modules, providing more efficient multi-turn optimization and domain adaptation capabilities compared to existing methods, advancing multimodal generation technology.
Novelty
GEMS is the first to apply an agent framework to multimodal generation, addressing complex instruction handling and downstream task adaptation through memory and skill modules.
Limitations
- In specific domain tasks, GEMS may require additional domain knowledge support.
- Expanding agent skills may increase system complexity.
Future Work
Future research directions include expanding the agent skill library, enhancing system adaptability and scalability, and exploring more downstream application scenarios.
AI Executive Summary
Multimodal generation has made significant progress in recent years, but challenges remain in handling complex instructions and specialized downstream tasks. The GEMS framework overcomes these limitations by introducing Agent Loop, Agent Memory, and Agent Skill. The Agent Loop improves generation quality through closed-loop optimization, Agent Memory provides persistent trajectory-level memory, reducing redundancy, and Agent Skill offers an extensible collection of domain-specific expertise.
In experiments, the GEMS framework excels across multiple mainstream and downstream tasks, notably enabling the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2. This demonstrates the effectiveness of the agent framework in extending model capabilities.
Despite significant performance improvements, GEMS may require additional domain knowledge support for specific tasks. Future research will focus on expanding the agent skill library and enhancing system adaptability and scalability.
Deep Analysis
Background
Multimodal generation has seen transformative growth, with representative works like GPT-Image and Nano Banana. These models excel in mainstream tasks but face challenges in complex instructions and specialized downstream tasks. Existing multi-agent systems achieve localized success but struggle to integrate with mainstream generative pipelines.
Core Problem
Multimodal generation faces challenges in handling complex instructions and specialized downstream tasks. Existing models struggle with intricate, multi-faceted instructions and lack adaptability in specific domain applications, limiting the widespread application of multimodal generation technology.
Innovation
The GEMS framework addresses complex instruction handling and downstream task adaptation by introducing Agent Loop, Agent Memory, and Agent Skill. Agent Loop improves generation quality through closed-loop optimization, Agent Memory provides persistent trajectory-level memory, reducing redundancy, and Agent Skill offers an extensible collection of domain-specific expertise.
Methodology
- �� Agent Loop: Improves generation quality through closed-loop optimization.
- �� Agent Memory: Provides persistent trajectory-level memory, reducing redundancy.
- �� Agent Skill: Offers an extensible collection of domain-specific expertise.
Experiments
Experiments were conducted across nine distinct tasks, including five mainstream benchmarks and four downstream tasks. The Z-Image-Turbo and Qwen-Image-2512 models were used for validation. Results show significant performance improvements across multiple tasks.
Results
GEMS achieves an average improvement of 14.22 points on mainstream tasks and 14.03 points on downstream tasks. Notably, on GenEval2, Z-Image-Turbo surpasses the state-of-the-art Nano Banana 2.
Applications
GEMS can be applied to complex instructions and specialized downstream tasks, such as creative drawing and academic illustration. Its extensible Agent Skill module provides broad application potential across different domains.
Limitations & Outlook
GEMS may require additional domain knowledge support for specific tasks. Expanding agent skills may increase system complexity. Future research will focus on expanding the agent skill library and enhancing system adaptability and scalability.
Plain Language Accessible to non-experts
Imagine a factory where workers have specific roles. GEMS is like a smart factory, with the Agent Loop as the production line optimizing product quality. Agent Memory is the archive room, recording experiences and data to avoid repetitive work. Agent Skill is the tool library, providing specialized tools and technical support. Through the collaboration of these modules, GEMS efficiently handles complex production tasks, enhancing product quality and diversity.
ELI14 Explained like you're 14
Imagine you're playing a super complex game where you manage multiple characters. GEMS is like a super smart game assistant, helping you optimize each character's actions. It has a memory system that records each character's actions and results, helping you avoid repeating mistakes. It also has a skill library that provides special skills when needed, helping you win the game. With these features, GEMS makes you a game master!
Glossary
GEMS (Agent-Native Multimodal Generation)
A multimodal generation framework that enhances generation quality through Agent Loop, Memory, and Skills.
GEMS is used to handle complex instructions and downstream tasks.
Agent Loop
A closed-loop optimization mechanism that improves generation quality through multiple iterations.
Used in GEMS to optimize the generation process.
Agent Memory
A persistent memory mechanism that records experiences and data during the generation process.
Used in GEMS to reduce redundancy and improve efficiency.
Agent Skill
An extensible skill library providing domain-specific expertise.
Used in GEMS to handle specialized downstream tasks.
Z-Image-Turbo
A lightweight 6B multimodal generation model.
Used in GEMS experiments to validate performance improvements.
Open Questions Unanswered questions from this research
- 1 How to expand the agent skill library without increasing complexity?
- 2 How to effectively integrate the GEMS framework across different domains?
- 3 How to further optimize the efficiency of Agent Memory?
Applications
Immediate Applications
Creative Drawing
GEMS can be used for creative drawing tasks, enhancing the quality and diversity of artistic expression through Agent Skills.
Long-term Vision
Intelligent Design Assistant
GEMS could evolve into an intelligent design assistant, supporting multi-domain design tasks, enhancing design efficiency and innovation.
Abstract
Recent multimodal generation models have achieved remarkable progress on general-purpose generation tasks, yet continue to struggle with complex instructions and specialized downstream tasks. Inspired by the success of advanced agent frameworks such as Claude Code, we propose \textbf{GEMS} (Agent-Native Multimodal \textbf{GE}neration with \textbf{M}emory and \textbf{S}kills), a framework that pushes beyond the inherent limitations of foundational models on both general and downstream tasks. GEMS is built upon three core components. Agent Loop introduces a structured multi-agent framework that iteratively improves generation quality through closed-loop optimization. Agent Memory provides a persistent, trajectory-level memory that hierarchically stores both factual states and compressed experiential summaries, enabling a global view of the optimization process while reducing redundancy. Agent Skill offers an extensible collection of domain-specific expertise with on-demand loading, allowing the system to effectively handle diverse downstream applications. Across five mainstream tasks and four downstream tasks, evaluated on multiple generative backends, GEMS consistently achieves significant performance gains. Most notably, it enables the lightweight 6B model Z-Image-Turbo to surpass the state-of-the-art Nano Banana 2 on GenEval2, demonstrating the effectiveness of agent harness in extending model capabilities beyond their original limits.