Measurement-Driven Sub-Network Selection for On-Premise Retrieval-Augmented Factory Agents
Measurement-driven sub-network selection using hardware-aware multi-objective optimization enables efficient deployment of compressed LLMs on industrial edge devices, recovering 95% of unpruned quality.
Key Findings
Methodology
This paper introduces a multi-stage pipeline combining weight-shared supernet training via sandwich-style in-place distillation with hardware-aware sub-network selection. The supernetwork, based on Llama-3.2, is elastically scaled along depth and width, producing an 11-point candidate grid. During training, importance-calibrated sampling ensures diverse sub-network training, with logits distilled from the full model. Post-training, sub-networks are extracted and evaluated on-device for throughput and quality. A multi-objective optimization integrates judged retrieval-augmented generation (RAG) quality and measured throughput, constrained by a general capability floor and memory budget. This approach enables dynamic, device-specific sub-network selection, optimizing deployment efficiency without retraining, and effectively recovers model performance after structural compression.
Key Results
- Structural pruning reduces model size by 13.7%, but retrieval-grounded distillation recovers 4.6% of the lost quality, achieving approximately 95% of the original unpruned model’s judged RAG quality. Experiments across three heterogeneous edge devices show that the selected sub-networks operate at 1.3 to 5 watts, maintaining high answer quality (RAG scores >0.77). The hardware-aware selection outperforms proxy-based or size-only rules, demonstrating superior balance between size, speed, and quality.
- Quantitative results indicate that the method achieves a significant reduction in energy consumption and latency, with the model running efficiently on platforms like Jetson Orin Nano, RevPi, and Arduino UNO Q. The approach ensures stable performance across devices, with throughput predictions within 2.5-3.1% error, enabling rapid device-specific calibration.
- Ablation studies confirm that the combination of supernetwork training, importance calibration, and hardware-based selection is crucial for optimal performance. The pipeline effectively handles multi-task, retrieval-augmented scenarios, and maintains robustness despite hardware heterogeneity.
Significance
This work addresses a critical challenge in deploying large language models in resource-constrained industrial environments. By integrating hardware measurements into the model selection process, it decouples model size from performance, enabling efficient, reliable on-premise deployment. The methodology advances the state of the art in model compression, multi-objective optimization, and retrieval-augmented generation, providing a practical framework for industrial AI applications. Its ability to recover most of the unpruned model’s quality while respecting hardware constraints marks a significant step toward scalable, edge-based intelligent factory agents, reducing reliance on cloud infrastructure and enhancing data sovereignty.
Technical Contribution
The paper introduces a novel multi-objective, hardware-grounded sub-network selection framework built upon a weight-shared supernetwork trained via sandwich-style in-place distillation. It innovatively combines elastic network scaling with device-specific throughput prediction, enabling dynamic sub-network assignment. The integration of retrieval-grounded distillation further specializes the selected sub-networks for factory-specific documentation, enhancing task accuracy. The approach effectively breaks the traditional size-performance correlation, providing a scalable, resource-aware deployment pipeline that can adapt to heterogeneous edge hardware without retraining.
Novelty
This research is the first to incorporate real hardware measurement metrics directly into the sub-network selection process for compressed large language models, moving beyond proxy-based or parameter-count heuristics. Its multi-objective optimization framework, balancing retrieval quality, throughput, and memory constraints, introduces a new paradigm for resource-aware model deployment. The combination of elastic supernet training, importance calibration, and retrieval-grounded distillation represents a significant innovation over prior work focused solely on structural pruning or knowledge distillation, enabling practical, on-device AI in industrial settings.
Limitations
- The accuracy of hardware measurement metrics heavily influences sub-network selection; measurement errors could lead to suboptimal choices, especially on highly heterogeneous devices.
- Retrieval quality depends on the external document corpus and retrieval system efficiency; poor retrieval recall or document quality can degrade answer relevance and fidelity.
- Training multiple candidate sub-networks and performing on-device evaluation incurs computational costs, which may limit scalability in extremely resource-constrained environments or during rapid deployment cycles.
Future Work
Future directions include developing adaptive, real-time sub-network adjustment mechanisms based on dynamic hardware states and workload variations. Extending the framework to incorporate multi-modal data, such as visual or sensor inputs, could broaden application scope. Additionally, integrating end-to-end performance tracing and feedback loops will refine sub-network selection, further reducing deployment costs. Research into more efficient training and evaluation pipelines will also enhance scalability and practical deployment in industrial settings.
AI Executive Summary
Deploying large language models (LLMs) in industrial environments faces significant challenges due to hardware resource constraints. Traditional compression methods like pruning and distillation often reduce model size at the expense of performance, and their reliance on parameter count as a proxy for quality fails to account for device-specific variations. This paper proposes a novel measurement-driven sub-network selection framework that combines a weight-shared supernetwork trained via sandwich-style in-place distillation with hardware-aware multi-objective optimization.
The core idea is to generate multiple candidate sub-networks with elastic scaling along depth and width, then evaluate each on-device for throughput and answer quality. By integrating judged retrieval-augmented generation (RAG) scores with measured device throughput, the system dynamically selects the most suitable sub-network for each hardware platform, respecting a predefined capability floor and memory budget. This approach effectively decouples model size from performance, enabling high-quality, resource-efficient deployment.
Experimental results on manufacturing manuals demonstrate that structural pruning reduces model size by 13.7%, while retrieval-grounded distillation recovers 4.6% of the lost quality, achieving approximately 95% of the original unpruned model’s performance. The selected sub-networks operate efficiently across heterogeneous edge devices, consuming only 1.3 to 5 watts, and maintain high retrieval and answer quality scores. The method surpasses proxy-based selection rules, providing a practical, scalable solution for industrial AI deployment.
This work significantly advances the deployment of large models in resource-constrained settings by integrating hardware feedback into the model selection process, enabling flexible, high-performance factory agents. Future research will explore dynamic sub-network adaptation, multi-modal integration, and end-to-end performance optimization, further pushing the boundaries of edge AI in manufacturing.
Deep Dive
Abstract
On-premise assistants can give factory workers conversational access to machine documentation, but models capable of the task rarely fit shop-floor hardware. We show that after structural compression and retrieval-grounded adaptation, model size is no longer a reliable predictor of adapted answer quality: general capability falls almost linearly with parameter count, while judged retrieval-augmented answer quality does not. We therefore treat deployment as a post-adaptation selection problem, committing one sub-network per device on judged answer quality and measured on-device throughput under a configurable general-capability floor and memory budget; rules that optimize size, speed, or quality alone each give up capability or throughput. A weight-shared supernetwork trained with sandwich-style in-place distillation keeps this selection inexpensive. In a manufacturing-manual case study, extraction costs 13.7 percent of the unpruned model's judged quality and retrieval-grounded distillation returns it to within 4.6 percent, recovering two thirds of the loss, and the same assistant runs across three heterogeneous edge tiers at 1.3 to 5 watts standby.