DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams
DiagramNet employs multi-stage training and a decoupled multi-agent workflow to achieve superior system-level diagram recognition, outperforming GPT-5 and industry benchmarks.
Key Findings
Methodology
This work introduces a hierarchical recognition framework combining multi-stage supervised fine-tuning, reinforcement learning, and LoRA adaptation. The core architecture involves a multi-agent workflow: a perception agent detects components via YOLOv11, a reasoning agent predicts connections using a 3B vision-language model, and a knowledge agent incorporates task-specific knowledge. The dataset, DiagramNet, contains 10,977 connection annotations and 15,515 chain-of-thought QA pairs across four subtasks: Listing, Localization, Connection, and Circuit QA. The training pipeline iteratively refines the model, leveraging structured rewards for relation accuracy and topology consistency, enabling robust generalization and transfer to downstream tasks.
Key Results
- On the DiagramNet benchmark, the integrated 3B model with multi-agent workflow achieved an overall score of 0.671, surpassing the 2025 EDA Elite Challenge winner and outperforming GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro by over 2× in end-to-end evaluation.
- The model effectively transferred to AMSBench with only 60 images for detector adaptation, achieving zero-shot connectivity reasoning comparable to GPT-5 and Claude-Sonnet-4, and surpassing the AMS state-of-the-art Netlistify.
- Ablation studies confirmed that the multi-agent workflow boosts performance significantly across models, with Gemini-2.5-Pro improving 128.7× on Task 1, demonstrating strong model-agnostic benefits.
Significance
This research addresses the critical challenge of automating understanding of complex, non-standardized system-level circuit diagrams, which are vital in chip architecture. By creating a high-quality multimodal dataset and a novel multi-agent recognition framework, it bridges the gap between visual perception and semantic reasoning, enabling more intelligent and efficient chip design workflows. The approach paves the way for future AI-driven automation in electronic design, reducing manual effort and accelerating innovation.
Technical Contribution
The paper introduces a hierarchical task decomposition for structured visual reasoning, integrating multi-stage training, reinforcement learning, and task-specific LoRA fine-tuning. The multi-agent architecture effectively isolates perception, relation inference, and knowledge reasoning, improving accuracy in dense, symbol-diverse diagrams. This combination of techniques offers a new paradigm for complex diagram understanding, surpassing prior end-to-end models and standard datasets.
Novelty
This is the first comprehensive multimodal dataset specifically targeting system-level diagrams, covering multiple tasks with detailed annotations. The multi-agent workflow, which decouples perception, relation prediction, and knowledge reasoning, represents a significant innovation over traditional monolithic models. The integration of reinforcement learning with structured rewards further enhances relational accuracy, setting new benchmarks in the field.
Limitations
- The model struggles with extremely complex or highly cluttered diagrams, especially when symbols are highly non-standardized or occluded, due to inherent visual ambiguity.
- Training relies on extensive annotated data, and while transferability is demonstrated, performance may degrade with fewer labeled samples in novel domains.
- Computational complexity remains high, particularly during multi-stage training and inference, requiring further optimization for real-time applications.
Future Work
Future directions include expanding the dataset to cover more diagram types and complexities, integrating self-supervised learning for better data efficiency, and optimizing inference speed. Exploring more scalable multi-agent architectures and applying this framework to broader diagram types, such as mechanical or architectural schematics, could further extend its industrial impact.
AI Executive Summary
System-level circuit diagrams serve as the blueprint for chip architecture, illustrating how modules like processors, memory controllers, and interfaces interconnect. Recognizing and understanding these diagrams automatically remains a significant challenge due to their non-standardized symbols, implicit connections, and semantic complexity. Existing methods rely heavily on template matching and domain-specific rules, which are limited by symbol diversity and lack of structured data.
To address these issues, the authors introduce DiagramNet, a comprehensive multimodal dataset comprising nearly 11,000 connection annotations and over 15,000 chain-of-thought QA pairs across four tasks: Listing, Localization, Connection, and Circuit QA. This dataset bridges the gap between generic vision-language models and domain-specific circuit understanding. Building on this, they propose a hierarchical recognition framework that decomposes the complex visual reasoning process into three stages: perception, relation inference, and knowledge reasoning.
The core technical innovation involves a multi-agent architecture: a perception agent detects components with YOLOv11, a reasoning agent predicts connections using a 3B vision-language model, and a knowledge agent incorporates task-specific knowledge via LoRA. The training pipeline combines supervised fine-tuning, reinforcement learning with structured rewards, and low-rank adaptation, resulting in a robust and transferable model.
Experimental results demonstrate that this approach outperforms industry benchmarks and state-of-the-art models, achieving an overall score of 0.671 on the DiagramNet benchmark, surpassing GPT-5 and industry champions. The model also generalizes effectively to AMSBench with minimal adaptation data, achieving zero-shot connectivity reasoning comparable to top models.
This work significantly advances automated understanding of complex, non-standardized system diagrams, offering a scalable solution for chip design automation. Future work will focus on dataset expansion, efficiency improvements, and broader diagram understanding, aiming to accelerate the digital transformation of electronic design processes.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂里有很多不同的机器和管道。这些机器代表不同的功能模块，管道代表它们之间的数据流。工厂的设计图就像系统级电路图，告诉你哪些机器连接在一起，怎么工作。以前的方法就像用模板匹配，只能识别一些标准的机器，但工厂里有各种不同的设备和连接方式，难以一一匹配。现在，这个新方法像一个聪明的工厂管理员，能看懂各种不同的机器和管道，自动识别它们的连接关系。它通过学习大量不同工厂的设计图，逐步掌握识别技巧，最终能在新工厂的图纸上快速找到所有机器和连接。这样一来，工厂设计图就变得像拼图一样容易理解，让工厂管理变得更智能、更高效。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，拼图上有很多不同的碎片，有的代表不同的机器，有的代表连接线。以前的拼图软件只能识别一些标准的碎片，遇到特别的碎片就认不出来。而现在，这个新方法像一个聪明的朋友，能看懂各种不同的碎片和线条，帮你把拼图拼得更快更准。它先用一个“眼睛”找到所有碎片的位置，再用“脑袋”推断哪些碎片应该连接在一起，最后用“知识”确认拼图的完整性。这个朋友学了很多拼图的技巧，能在不同的拼图上都表现出色。这样一来，拼图变得简单多了，你可以专注于玩乐，而不用担心拼错。它让复杂的电路图变得像拼图一样容易理解，未来可以帮工程师们更快设计芯片，节省时间和精力！
Abstract
System-level diagrams encode the architectural blueprint of chip design, specifying module functions, dataflows, and interface protocols. However, non-standardized symbols and the scarcity of structured training data hinder existing multimodal large language models (MLLMs) from recognizing these diagrams. To address this gap, we introduce DiagramNet, the first multimodal dataset for system-level diagrams, comprising 10,977 connection annotations and 15,515 chain-of-thought QA pairs across four tasks: Listing, Localization, Connection, and Circuit QA. Building on this dataset, we propose a progressive training pipeline together with a decoupled multi-agent workflow that decomposes complex visual reasoning into Perception, Reasoning, and Knowledge stages. On the DiagramNet benchmark, integrating our 3B-parameter model with the proposed workflow surpasses the 2025 EDA Elite Challenge winner and outperforms GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro by over 2x in end-to-end evaluation. Notably, the workflow generalizes beyond our model, boosting Task 1 performance by 128.7x for Gemini-2.5-Pro and 12.4x for GPT-5. Furthermore, with only 60 images for detector adaptation, the method transfers effectively to AMSBench, achieving zero-shot connectivity reasoning on par with GPT-5 and Claude-Sonnet-4 while surpassing the AMS state-of-the-art method Netlistify.