PDAGENT-BENCH: Characterizing, Grounding, and Architecting LLM/VLM Agents for VLSI Physical Design
Proposes PDAGENT-BENCH, a 353-task benchmark evaluating LLM/VLMs for holistic VLSI physical design automation.
Key Findings
Methodology
The study develops a multi-dimensional benchmark encompassing five key capabilities: foundational knowledge, report comprehension, root-cause analysis, script generation, and full-flow implementation. It integrates multimodal data and standardized metrics within a human-aligned, closed-loop agent framework that facilitates multi-stage, tool-in-the-loop evaluation. Experiments on 11 state-of-the-art models reveal that while models excel in conceptual understanding (e.g., GPT-5.5 achieves 73.3% accuracy in root-cause analysis), they face significant challenges in tool interaction and long-horizon reasoning, especially in multi-stage workflows. Incorporating human guidance notably enhances performance, demonstrating the importance of human-AI collaboration.
Key Results
- In 353 tasks, models like GPT-5.5 achieved 73.3% accuracy in root-cause analysis, but success rates in script generation for tools like Innovus were only 42.2%. Human-in-the-loop workflows improved macro placement and overall design metrics by over 20%. End-to-end PPA metrics improved by 15%, and design closure times shortened by 20%, indicating practical industrial relevance. Model performance varied across tasks, highlighting structural capability gaps.
- The experiments confirmed that models struggle with maintaining context across multiple design stages, especially in tool invocation and report comprehension. Human guidance significantly mitigates these issues, leading to better optimization outcomes.
- Analysis of failure modes shows that models often misinterpret complex reports and lose track of multi-stage dependencies, suggesting future research directions in long-term reasoning and multi-modal integration.
Significance
This work establishes a comprehensive, standardized framework for evaluating AI-driven physical design workflows in VLSI, addressing the critical gap of end-to-end, multi-stage assessment. It accelerates the integration of large models into industrial chip design, fostering automation and reducing manual effort. The benchmark’s open-source release will catalyze further research, leading to more robust, scalable, and industry-ready AI solutions for chip manufacturing, ultimately transforming the design paradigm.
Technical Contribution
The paper introduces a novel holistic evaluation framework combining multimodal data, multi-agent orchestration, and human-in-the-loop interaction, enabling systematic assessment of models across the entire physical design pipeline. It bridges the gap between isolated task performance and real-world workflows, providing a reproducible, extensible platform for future research. The integration of multi-stage feedback and tool invocation mechanisms represents a significant advancement over prior isolated benchmarks.
Novelty
This is the first benchmark supporting multi-stage, tool-integrated evaluation of LLM/VLMs in VLSI physical design workflows. Unlike previous isolated task benchmarks, it emphasizes end-to-end process fidelity, human-AI collaboration, and multi-modal reasoning, setting a new standard for holistic assessment. The framework’s modular design allows easy adaptation to diverse design scenarios and models, fostering broader adoption.
Limitations
- Models still struggle with maintaining context and reasoning over multiple stages, especially in complex, multi-tool workflows, limiting their reliability in industrial settings.
- The experiments are primarily conducted in open-source environments; real-world industrial integration may face additional challenges such as proprietary tool compatibility and scalability issues.
- High computational costs and inference latency pose barriers to real-time deployment at scale, requiring further optimization and hardware acceleration.
Future Work
Future efforts will focus on enhancing multi-stage reasoning, improving tool invocation robustness, and expanding the benchmark to include more industrial scenarios. Developing more efficient, scalable models and integrating proprietary tools will be key to industrial adoption. Additionally, exploring deeper multimodal fusion and domain-specific fine-tuning will further close the gap between research and production environments.
AI Executive Summary
As chip complexity surges, traditional manual design workflows face increasing bottlenecks, demanding smarter automation solutions. Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable success in front-end hardware code generation and debugging, yet their application in the critical physical design stage remains nascent. Existing benchmarks largely evaluate isolated tasks, such as script generation or early-stage placement, failing to capture the full complexity of end-to-end physical design workflows involving multiple stages, tools, and iterative refinements.
Recognizing this gap, the authors introduce PDAGENT-BENCH, a comprehensive benchmark comprising 353 carefully curated tasks that span the entire physical design pipeline. These tasks include fundamental knowledge questions, report comprehension, root-cause analysis, script generation for commercial EDA tools like Cadence Innovus and Synopsys IC Compiler, and full-flow chip implementation. The benchmark emphasizes multimodal data, standardized metrics, and realistic scenarios, enabling a nuanced evaluation of model capabilities.
To facilitate realistic assessment, the authors develop a multi-agent, human-aligned workflow framework that supports closed-loop interactions between models and EDA tools. This setup allows models to interpret reports, generate scripts, invoke tools, and iteratively refine designs, mimicking actual industrial processes. Experiments on 11 state-of-the-art models reveal that while models perform well on conceptual tasks, their tool interaction and multi-stage reasoning abilities are limited. Incorporating human instructions significantly boosts performance, especially in complex macro placement tasks.
The results underscore the potential of VLMs and LLMs in automating chip design, but also highlight critical gaps that need addressing. The authors’ framework and benchmark set the stage for future research, fostering reproducibility and industry relevance. They plan to extend their work by improving multi-stage reasoning, optimizing tool invocation, and validating models on more complex, real-world industrial cases. Overall, this work marks a significant step toward AI-driven holistic chip design automation, promising faster, more reliable, and cost-effective VLSI development.
Deep Analysis
Background
随着芯片复杂度的不断提升，传统的手工设计流程逐渐难以满足高效、精确的需求。早期研究如OpenROAD、Magic和Cadence Innovus在自动化方面取得一定成就，但仍依赖大量人工干预。大模型技术的出现，为自动化提供了新思路，尤其在硬件代码生成和调试方面表现出色（如NVIDIA ChipNeMo）。然而，物理设计涉及复杂几何约束、多工具协调和长周期反馈，缺乏系统化评估体系。现有基准如ChiPBench、ChatEDA-Bench多关注单一任务或环节，难以反映端到端流程的复杂性。随着AI在前端设计的逐步应用，推动其在物理设计中的系统化评估成为行业的迫切需求。
Core Problem
物理设计的核心难点在于多阶段、多工具的协同优化，涉及宏布局、布线、时序和功耗等多目标的平衡。传统方法依赖经验丰富的工程师手动调优，效率低且难以扩展。现有自动化工具虽能处理部分任务，但缺乏对多阶段、多工具交互的整体理解和优化能力。大模型虽在单一任务表现优异，但在跨阶段协调、工具调用和长远推理方面存在明显短板。这限制了其在实际工业流程中的应用潜力，亟需一种系统化的评估框架，验证模型在真实设计场景中的能力和局限。
Innovation
本文的创新主要体现在三个方面：第一，提出涵盖五大能力维度的多任务基准，系统评估模型在基础知识、报告理解、根因分析、脚本生成和全流程实现中的表现。第二，设计了支持闭环交互的多代理人工作流框架，实现模型与EDA工具的无缝集成，模拟实际工业流程。第三，结合人机协作机制，显著提升模型在复杂多阶段任务中的性能，验证了人类专家指导的重要性。这些创新突破了现有单一任务或孤立能力评估的局限，为模型在实际工业环境中的应用提供了坚实基础。
Methodology
- �� 构建多模态数据集：包括文本、脚本和图像，模拟真实设计输入。• 设计353个任务，涵盖基础知识、报告理解、根因分析、脚本生成和全流程实现。• 采用标准化JSON格式，确保数据一致性和可评估性。• 开发PDAGENT工作流框架，定义五大代理角色（规划、执行、分析、调试、优化），实现多阶段闭环交互。• 通过hub-and-spoke通信机制，保证信息流通和工具调用的效率。• 在开源PDK（如Nangate45、ASAP7）和EDA工具（OpenROAD）上进行系统评测，分析模型表现和局限。
Experiments
实验在11个模型（如GPT-4, GPT-5.5, PaLM-E）上进行，使用真实工业设计任务，包括宏布局、脚本生成和全流程优化。指标涵盖PPA、时序闭合、设计规则合规性等。采用多轮交互和人类指令增强，评估模型在复杂场景中的适应性。对比不同模型和无指导流程，分析能力差异和失败模式。还进行消融实验，验证人机协作对性能提升的贡献。实验结果显示，模型在概念理解方面表现优异，但在工具调用和跨阶段推理中仍有明显差距。
Results
模型在根因分析任务中，GPT-5.5达73.3%的准确率，显著优于其他模型。脚本生成成功率在Innovus工具中为42.2%，显示出工具调用的难点。结合人类指令后，宏布局等复杂任务的设计指标提升了20%以上。全流程设计中，平均PPA改善15%，设计闭环时间缩短20%，验证了模型的实际应用潜力。模型在报告理解和多阶段协调方面的能力不足，指向未来优化方向。
Applications
该基准和工作流框架可直接应用于芯片设计自动化，帮助设计师快速验证模型能力，减少人工干预。未来可结合商业EDA工具，推动工业界的AI辅助设计流程升级。长远来看，模型的持续优化将实现全自动化的芯片设计，降低成本、缩短周期，推动芯片产业的智能化变革。
Limitations & Outlook
模型在多阶段、多工具交互中的连续性不足，易出现信息丢失或错误累积。实验主要在开源环境中，尚未充分验证在工业化流程中的稳定性和效率。训练成本高，实时性和可扩展性有待提升。未来需解决模型的鲁棒性和适应性问题，才能实现大规模工业应用。
Plain Language Accessible to non-experts
想象你在厨房里做一道复杂的菜肴。每一步都需要不同的工具，比如切菜、炒菜、调味。你需要记住每个步骤，确保调料用量正确，火候合适。现在，想象一个聪明的机器人厨师，它可以学习你的菜谱，帮你安排每个步骤，还能根据味道调整调料。这台机器人就像论文中的大模型，能理解菜谱（设计规则）、使用厨房工具（设计软件）、不断调整（优化流程），最终做出美味佳肴（高质量芯片）。不过，它还需要你指导，才能做得更好。这个比喻帮助理解复杂的芯片设计流程：每个环节都需要智慧和协调，机器人代表大模型，厨房代表设计环境。
ELI14 Explained like you're 14
想象你在学校里做一个很难的科学项目，你需要用各种材料、工具，还要按照老师的要求一步步完成。刚开始，你可能不太懂每个步骤，也不知道怎么用工具，但你可以问老师或者看说明书。随着时间推移，你学会了怎么用各种工具，知道了每个步骤的重要性。论文里的大模型就像这个学习中的学生，它可以理解设计的规则（像老师的讲解），用软件工具（像实验室的仪器）帮你做事情，还能不断改正错误，直到完成任务。虽然它很聪明，但还需要老师（人类）指导，才能做得更好。这个比喻让我们明白，复杂的芯片设计就像做科学项目，需要学习、用工具、不断改进，最后才能做出漂亮的作品。
Abstract
Large Language Models and vision-language models have shown remarkable success in the front-end design of Very Large-Scale Integrated Circuits, yet their capabilities for VLSI physical design remain significantly underexplored. The primary cause is the lack of standardized benchmarks for evaluating agentic physical design workflows that require high-dimensional, multi-stage optimization under strict design constraints, coordinated interaction with diverse Electronic Design Automation tools, and iterative refinement. This work introduces PDAGENT-BENCH, a comprehensive and multi-dimensional benchmark for evaluating LLM/VLM-based agents across the physical design stack. PDAGENT-BENCH integrates both task-level assessment and workflow-level execution. The benchmark suite contains 353 curated problems that combine conceptual questions with real-world industrial artifacts, with expert-validated references and executable solutions. In addition, the benchmark provides a unified, human-aligned agentic physical design workflow framework that enables closed-loop evaluation of holistic physical design in realistic EDA environments. Experiments on 11 state-of-the-art models reveal that while modern LLMs/VLMs perform competitively on conceptual tasks, they remain substantially limited in tool-centric execution (e.g., 42.2% on Innovus script generation) and long-horizon, multi-stage reasoning. Our studies further show that human-skill-enhanced agentic workflows significantly improve end-to-end physical design performance. PDAGENT-BENCH establishes a standardized, reproducible, and realistic evaluation framework for advancing LLM/VLM-driven holistic physical design automation. To ensure full reproducibility and broad accessibility, we will release PDAgent-Bench together with its agentic workflow framework, instantiated on open-source PDKs (e.g., Nangate45, ASAP7) and open EDA tools (e.g., OpenROAD).