HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
HR-NAS integrates lightweight transformers and multi-scale search space to optimize high-resolution dense prediction networks.
Key Findings
Methodology
HR-NAS constructs a multi-branch search space combining convolutional and transformer modules, enabling multi-scale feature fusion. It employs a dynamic complexity-adjustable lightweight transformer to encode global context efficiently. A progressive pruning strategy is used to optimize the architecture by importance-based feature unit removal, balancing performance and efficiency. The search space includes various resolutions, supporting multi-task adaptation. During training, importance factors and resource penalties guide the automatic selection of key units, ensuring optimal trade-offs across tasks.
Key Results
- On Cityscapes semantic segmentation, HR-NAS surpasses SqueezeNAS with 45.9% efficiency gain, reaching 78.2% mIoU at 325M FLOPs. For COCO human pose estimation, it achieves 76.5 AP with low computational cost. On ImageNet classification, it attains 75.7% Top-1 accuracy with only 5.5M parameters, demonstrating excellent performance-efficiency balance. Multi-task experiments confirm strong generalization, outperforming manual and existing NAS models.
Significance
This work advances NAS by integrating high-resolution features and global context encoding, addressing the limitations of prior classification-focused methods. The combination of lightweight transformers and multi-scale fusion enables dense prediction tasks to benefit from automatic architecture design, significantly improving accuracy and efficiency. Its resource-aware search allows deployment on diverse hardware, promoting applications in autonomous driving, robotics, and medical imaging. Overall, HR-NAS sets a new benchmark for automated high-resolution dense prediction network design.
Technical Contribution
The paper introduces a novel multi-scale search space with integrated lightweight transformer modules, enabling efficient global context encoding within resource constraints. It proposes a dynamic complexity regulation mechanism for transformers, combined with a progressive importance-based pruning strategy, to automatically optimize architectures. The multi-branch design supports multi-resolution feature fusion, enhancing both local detail and global understanding. These innovations collectively push the frontier of NAS for dense prediction, balancing performance and computational cost effectively.
Novelty
This is the first work to embed a lightweight, dynamically adjustable transformer into a resource-constrained NAS search space specifically for dense prediction tasks. It innovates by combining multi-scale convolutional and transformer modules within a unified search framework, enabling adaptive multi-resolution feature fusion. The progressive pruning approach further distinguishes it from prior methods, allowing automatic architecture customization for diverse tasks and hardware budgets, leading to superior performance with lower computational costs.
Limitations
- The model’s performance drops under extremely low FLOPs budgets, limiting applicability in ultra-lightweight scenarios. The dynamic complexity regulation of transformers still requires manual tuning, which could hinder widespread adoption. The multi-scale architecture adds complexity, potentially impacting real-time inference speed. Future work should focus on further simplifying transformer modules and optimizing search algorithms for broader hardware compatibility.
Future Work
Future directions include designing more efficient transformer variants tailored for resource-limited devices, exploring hardware-aware search strategies, and extending the framework to video and 3D dense prediction tasks. Incorporating self-supervised learning could enhance model robustness and generalization. Additionally, integrating latency and energy-aware metrics into the search process will facilitate deployment in real-world systems, broadening the impact of this approach.
AI Executive Summary
High-resolution representations are vital for dense prediction tasks such as segmentation, detection, and pose estimation. However, existing neural architecture search (NAS) methods predominantly focus on image classification, neglecting the importance of multi-scale features and global context in dense tasks. HR-NAS addresses this gap by proposing a novel framework that combines a multi-branch architecture inspired by HRNet with a lightweight, dynamically adjustable transformer module. This design enables the network to encode multiscale contextual information effectively while maintaining high-resolution features throughout the network.
The core innovation lies in integrating a resource-aware, fine-grained search strategy that employs importance factors and resource penalties to automatically prune redundant channels and transformer tokens. This approach results in architectures that are optimized for various tasks and computational budgets, achieving superior performance-efficiency trade-offs. Extensive experiments on benchmarks such as Cityscapes, COCO, and ImageNet demonstrate that HR-NAS outperforms state-of-the-art methods, including SqueezeNAS, with a 45.9% efficiency improvement in semantic segmentation and comparable or better accuracy in pose estimation and classification.
The significance of this work extends beyond mere performance metrics. It introduces a paradigm shift in NAS for dense prediction, emphasizing the importance of high-resolution feature preservation and global context encoding. By enabling automatic, resource-aware architecture customization, HR-NAS paves the way for deploying advanced dense prediction models in real-world applications like autonomous vehicles, robotics, and medical imaging, where efficiency and accuracy are both critical.
Despite its strengths, the method faces limitations such as performance degradation under extremely low FLOPs budgets and the need for manual tuning of transformer complexity. Future research will focus on further optimizing transformer efficiency, expanding multi-task capabilities, and integrating hardware-aware metrics to facilitate broader adoption in industry settings.
Deep Analysis
Background
深度学习在图像识别、目标检测等任务中取得巨大成功，但高分辨率特征的有效利用仍是瓶颈。传统NAS多专注于分类任务，忽视了密集预测对多尺度特征和全局上下文的需求。HRNet等多分支架构成功保持高分辨率，但缺乏自动化搜索优化。Transformer引入带来全局信息，但计算成本高，限制其在资源有限场景中的应用。近年来，结合多尺度卷积和Transformer的研究逐渐兴起，旨在提升密集预测性能。
Core Problem
现有NAS方法多未充分考虑高分辨率特征的保持与多尺度融合，导致在密集预测任务中的表现不足。深度模型在细节与全局信息之间存在权衡，缺少自动化调优手段。Transformer虽能捕获全局依赖，但高计算成本限制其应用。如何设计既高效又能充分利用多尺度信息的网络架构，成为亟待解决的难题。
Innovation
本研究提出多尺度搜索空间，结合轻量级Transformer与卷积模块，支持多分支特征融合。引入动态调节复杂度的Transformer，降低计算成本，增强全局感知。采用渐进式剪枝策略，自动筛选关键特征单元，优化模型结构。多尺度融合架构支持多任务适应，突破了Transformer在资源受限场景中的应用瓶颈。整体架构实现了性能与效率的最佳平衡。
Methodology
- �� 设计多尺度、多分支搜索空间，融合卷积和Transformer模块。
- �� 构建轻量Transformer，采用投影降低复杂度，结合空间位置编码。
- �� 在每个分支堆叠搜索块，实现多尺度特征融合。
- �� 引入重要性系数与资源惩罚，自动筛选关键单元。
- �� 采用渐进式剪枝，动态调节模型结构。
- �� 训练过程中优化结构参数，支持多任务泛化。
- �� 最终模型可直接应用，无需微调。
Experiments
在Cityscapes、COCO、ImageNet等数据集上验证。采用多任务训练策略，设置不同资源预算。与SOTA方法对比，评估mIoU、AP、Top-1准确率。通过消融实验验证Transformer模块、剪枝策略的贡献。参数设置包括参数量、FLOPs、训练轮次，确保公平性。
Results
在Cityscapes语义分割中，模型FLOPs为325M，达78.2% mIoU，优于SqueezeNAS，效率提升45.9%。COCO人体姿态估计中，AP达76.5%，在保持低计算成本基础上表现优异。ImageNet分类中，Top-1准确率达75.7%，参数仅5.5M，展现出极佳的性能效率比。多任务验证显示模型具有良好的泛化能力，优于多数手工设计网络。
Applications
该架构适用于自动驾驶、无人机、机器人等需要高精度密集预测的场景。模型可部署在边缘设备，支持实时处理。其多尺度融合能力也适合医学影像、遥感等领域的细节分析。未来可结合硬件感知优化，推动工业界的广泛应用。
Limitations & Outlook
模型在极低FLOPs预算下性能下降明显，难以满足某些极端应用需求。Transformer的复杂度调节仍需手动调参，影响推广。多尺度架构增加模型复杂度，可能影响推理速度。未来需优化Transformer设计，提升在边缘设备上的适应性。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材就像图片中的不同细节和信息。传统方法就像用单一锅煮所有食材，容易煮糊或不熟。而HR-NAS就像用多个小锅同时煮不同的食材，然后再把它们融合在一起，确保每个细节都能充分展现。轻量级Transformer就像一个聪明的厨师，能快速理解所有食材的关系，帮你调配出最美味的菜肴。这种方法让你在有限时间和工具下，也能做出既细腻又丰富的菜肴，适合各种厨房（任务）需求。
ELI14 Explained like you're 14
想象你在学校的科学实验室里做实验，实验需要观察很多细节，比如细菌的形状和运动。以前的电脑模型就像用放大镜看一部分，容易漏掉整体情况。而HR-NAS就像用一个智能望远镜，不仅能看到细节，还能同时看到整体。它用很多小工具（多分支）把不同的细节融合在一起，还用一个聪明的助手（轻量Transformer）帮忙看全局。这样，不管你是要分类、检测还是识别细节，这个系统都能帮你快速找到最重要的部分，做出准确的判断。它既聪明又快，能在有限的时间和资源里完成复杂的任务，就像一个超级助手一样！
Abstract
High-resolution representations (HR) are essential for dense prediction tasks such as segmentation, detection, and pose estimation. Learning HR representations is typically ignored in previous Neural Architecture Search (NAS) methods that focus on image classification. This work proposes a novel NAS method, called HR-NAS, which is able to find efficient and accurate networks for different tasks, by effectively encoding multiscale contextual information while maintaining high-resolution representations. In HR-NAS, we renovate the NAS search space as well as its searching strategy. To better encode multiscale image contexts in the search space of HR-NAS, we first carefully design a lightweight transformer, whose computational complexity can be dynamically changed with respect to different objective functions and computation budgets. To maintain high-resolution representations of the learned networks, HR-NAS adopts a multi-branch architecture that provides convolutional encoding of multiple feature resolutions, inspired by HRNet. Last, we proposed an efficient fine-grained search strategy to train HR-NAS, which effectively explores the search space, and finds optimal architectures given various tasks and computation resources. HR-NAS is capable of achieving state-of-the-art trade-offs between performance and FLOPs for three dense prediction tasks and an image classification task, given only small computational budgets. For example, HR-NAS surpasses SqueezeNAS that is specially designed for semantic segmentation while improving efficiency by 45.9%. Code is available at https://github.com/dingmyu/HR-NAS