AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

The evolution of multimodal content generation has seen rapid advancements with models like GPT-4, Perceiver, and CoCa, which integrate text, images, and other modalities to produce coherent outputs. Early systems focused on single-modality tasks, such as language modeling or image synthesis, but recent efforts aim to unify these capabilities for complex applications like automatic report generation, multimedia summarization, and scientific visualization. Despite progress, existing systems predominantly operate as static models, lacking mechanisms for continuous self-improvement. Prior works such as TextGrad, DSPy, and GEPA have explored component-level optimization, while systems like STOP, GPTSwarm, and AFlow have emphasized workflow search. However, these approaches do not address the recursive, long-term optimization of entire content generation systems grounded in human preferences. The challenge remains in designing systems that can adaptively refine their architecture and strategies over multiple tasks and iterations, especially in high-stakes domains like scientific publishing, where fidelity, clarity, and visual appeal are paramount. The paper situates itself within this landscape, proposing a novel meta-harness framework that enables persistent, recursive system evolution, validated through the academic paper-to-poster task, a representative multimodal design problem.

Traditional content generation pipelines are limited by their static nature, often producing outputs that are either faithful but unrefined or visually appealing but lacking in fidelity. In complex multimodal tasks, this trade-off is exacerbated by the inability to incorporate feedback effectively across multiple stages. The core problem addressed by this work is how to develop a system that can not only generate high-quality artifacts but also learn from its own outputs to improve over time. Specifically, in the context of academic paper-to-poster conversion, the challenge lies in balancing source fidelity, dense scientific communication, visual coherence, and usability of the final artifact. Existing methods lack a systematic mechanism for persistent knowledge accumulation and self-directed improvement, leading to suboptimal performance and manual intervention requirements. The key difficulty is designing a framework that can leverage multi-round feedback, analyze recurrent failures, and implement targeted system updates without overfitting or catastrophic forgetting, thereby enabling long-term autonomous evolution.

The primary innovation of AutoDesign is the introduction of a meta-harness optimization framework that treats system design as a recursive, long-horizon problem. Unlike prior work focusing on static models or single-pass optimization, AutoDesign employs a hierarchical feedback loop architecture:

• �� The inner loop involves a collaborative content generator (designer) and critic, which iteratively produce and refine artifacts based on rule-based and visual feedback.
• �� The outer loop analyzes multi-round trajectories and evaluation scores to identify systemic failures and propose bounded component updates.
• �� The updates are generated by a meta-optimizer, which employs Bayesian optimization or reinforcement learning, ensuring performance gains on training data without regressions on validation data.
• �� The system decomposes the design harness into five modules—context, tools, runtime, orchestration, and evaluation—allowing targeted, interpretable updates.
• �� The evaluation protocol (PosterBench) comprehensively assesses multiple quality dimensions, aligning system improvements with human preferences.

This approach enables the system to continually learn from its outputs, forming a self-improving loop that surpasses traditional static systems, especially in complex, multi-step design tasks.

• �� The design harness (H) is modular, comprising five functional components: context/memory, tools/specifications, execution runtime, orchestration, and evaluation/feedback.
• �� The inner loop involves a designer model (e.g., GPT-4) generating an artifact y and a critic providing feedback f, iteratively refining the artifact until a satisfactory version is produced.
• �� The outer loop executes multiple rounds of system deployment on diverse tasks, collecting trajectories τ and evaluation scores s based on a comprehensive evaluator Rmeta, which combines rule-based checks and vision-language model judgments.
• �� The optimizer P analyzes these trajectories and scores, identifying recurrent failure modes and proposing bounded component updates H′ based on observed deficiencies.
• �� Updates are accepted only if they improve performance on the training set without degrading validation results, using an acceptance gate.
• �� The process repeats over multiple iterations, gradually evolving the design system into a self-improving, human-aligned content generator.
• �� The entire framework is instantiated in the academic paper-to-poster task, demonstrating end-to-end automation from source ingestion to final visualization.

• �� The evaluation employs PosterBench, a comprehensive benchmark with 100 papers spanning five disciplines and a smaller 10-paper subset for rapid testing.
• �� Baselines include static systems like Claude Design and various configurations of AutoDesign with different system modules.
• �� Metrics encompass overall PosterBench score, fidelity, coverage, density, visual evidence, layout, readability, and aesthetics, evaluated through rule-based checks, vision-language models, and human judgments.
• �� Hyperparameters such as trajectory sampling size, update thresholds, and module-specific learning rates are tuned for stability.
• �� Ablation studies compare the impact of individual modules and feedback mechanisms, confirming the importance of recursive, bounded updates.
• �� Results show that AutoDesign consistently outperforms baselines, with the best configuration reaching 78.32 points, a significant margin over existing solutions.

• �� AutoDesign achieved a PosterBench main track score of 78.32, outperforming Claude Design by 7.45 points, demonstrating its superior ability to generate faithful, dense, and visually appealing posters.
• �� Across seven configurations, integrating the learned DesignHarness improved scores from 54.99 to 67.39, confirming the robustness of the recursive optimization process.
• �� Fully autonomous runs completed within 40 minutes, executing 253 tool calls and 11 editing turns at a cost below $3, producing artifacts comparable to human-designed posters.
• �� Human preference surveys indicated that 64% of evaluators favored AutoDesign outputs, especially in visual coherence and content density, validating its practical effectiveness.

• �� Immediate: Researchers can use AutoDesign to automate poster creation, reducing manual effort and accelerating dissemination of scientific results.
• �� Educational institutions can generate teaching materials, visual summaries, and interactive content with minimal human input.
• �� Industry sectors like marketing and media can leverage the framework for rapid multimodal content production, enhancing creativity and productivity.
• �� Long-term: The framework can evolve into a general-purpose autonomous content creation platform, capable of real-time, personalized multimedia generation, transforming creative industries and knowledge dissemination processes.

• �� The current validation is limited to academic paper-to-poster tasks; adaptation to other domains like video synthesis or interactive design remains untested.
• �� The recursive optimization process incurs high computational costs, especially for larger-scale or real-time applications.
• �� Fully autonomous operation may still face robustness challenges, such as handling ambiguous feedback or conflicting evaluation signals, requiring further research into explainability and safety mechanisms.