Mobile-Former: Bridging MobileNet and Transformer
Proposed Mobile-Former combines MobileNet and Transformer with few learnable tokens, achieving high accuracy at low FLOPs.
Key Findings
Methodology
Mobile-Former employs a parallel design, integrating MobileNet for local feature extraction and a lightweight Transformer for global interaction. A bidirectional cross-attention mechanism connects them, with only a few learnable tokens (e.g., 6) representing global priors, drastically reducing computational cost. The Transformer component uses multi-head attention with minimal tokens, while the cross-attention at the Mobile bottleneck enables efficient feature fusion. Dynamic ReLU activation enhances non-linearity. Extensive experiments on ImageNet, COCO, and DETR-based detection demonstrate superior performance and efficiency over MobileNetV3, with significant reductions in FLOPs and parameters.
Key Results
- On ImageNet classification, Mobile-Former-294M achieves 77.9% top-1 accuracy at 294M FLOPs, outperforming MobileNetV3 by 1.3% while saving 17% of computations.
- In RetinaNet detection, it surpasses MobileNetV3 by 8.6 AP points.
- Replacing DETR backbone with Mobile-Former yields 1.1 AP improvement, with 52% fewer FLOPs and 36% fewer parameters, demonstrating high efficiency.
Significance
This work advances lightweight vision models by effectively integrating local and global features within a parallel framework. It addresses the challenge of balancing accuracy and efficiency, especially critical for edge devices and real-time applications. The architecture sets a new benchmark for low-cost high-performance models, inspiring future research on hybrid designs and global feature modeling under resource constraints.
Technical Contribution
The key innovation is the parallel architecture with a bidirectional lightweight cross-attention bridge, enabling efficient fusion of local features and global priors. The minimal token Transformer reduces complexity, while dynamic ReLU boosts non-linear capacity. The design achieves state-of-the-art results across classification and detection tasks, demonstrating a new paradigm for resource-efficient vision models.
Novelty
This is the first work to parallelize MobileNet and Transformer with a few learnable tokens for global context, employing a lightweight cross-attention mechanism. Unlike previous serial or stacked fusion methods, this approach offers a highly efficient and effective way to incorporate global information in low FLOP regimes.
Limitations
- Performance drops in extremely low FLOP regimes (e.g., under 10M FLOPs), indicating the need for further global feature compression.
- Transferability to video or multi-modal tasks remains untested, requiring additional adaptation.
- Adding more tokens improves accuracy slightly but increases complexity; optimal token count needs further exploration.
Future Work
Future directions include optimizing token learning strategies, exploring multi-scale global features, and extending the architecture to video and multi-modal applications. Hardware-aware design and deployment on edge devices will also be prioritized to enhance real-world usability.
AI Executive Summary
Mobile-Former introduces a novel parallel architecture that unites MobileNet's local feature extraction with a lightweight Transformer for global interaction. Its core innovation lies in a bidirectional, low-cost cross-attention bridge that connects a few learnable global tokens with local features, enabling efficient global context modeling without significant computational overhead.
This design addresses the longstanding challenge of integrating global information in resource-constrained environments. Unlike traditional Transformers that rely on dense attention over many tokens, Mobile-Former uses only a handful of tokens, drastically reducing complexity. Extensive experiments on ImageNet demonstrate that Mobile-Former-294M achieves 77.9% top-1 accuracy, surpassing MobileNetV3 by 1.3%, while consuming 17% less FLOPs. In object detection, it outperforms MobileNetV3 by 8.6 AP points in RetinaNet and improves DETR-based end-to-end detection by 1.1 AP with nearly half the computational cost.
The architecture's efficiency stems from the parallel fusion of local and global features, leveraging dynamic ReLU for enhanced non-linearity. Its flexibility allows deployment across classification, detection, and potentially other vision tasks, especially on edge devices where computational resources are limited. The results set a new benchmark for lightweight models, demonstrating that effective global modeling is feasible under strict efficiency constraints.
Looking ahead, the authors plan to refine token learning strategies, extend the framework to video and multi-modal tasks, and optimize hardware deployment. Despite its successes, the model still faces challenges in extremely low FLOP regimes and multi-task generalization, which will be focal points for future research. Overall, Mobile-Former paves a promising path toward high-performance, resource-efficient vision systems.
Deep Analysis
Background
Recent advances in deep learning have seen CNNs like MobileNet series excel in mobile scenarios due to their efficiency. Meanwhile, Vision Transformers (ViT) and variants such as Swin and DeiT have demonstrated superior performance by modeling global relationships but at high computational costs. Efforts to combine CNNs and Transformers aim to leverage their respective strengths, with serial stacking or hybrid modules. However, these approaches often face challenges in balancing efficiency and accuracy, especially under strict FLOP constraints. The need for models that can efficiently encode both local details and global context remains critical for deploying AI in real-time, resource-limited environments.
Core Problem
The core challenge is designing a neural network that can simultaneously achieve high accuracy and low computational cost. Existing lightweight CNNs excel at local feature extraction but struggle with global context, while Transformers provide global modeling but are computationally expensive. Combining these effectively without incurring prohibitive costs is difficult. The bottleneck lies in how to fuse local and global information efficiently, especially when the number of tokens or parameters is limited. Achieving this balance is essential for applications like mobile vision, autonomous systems, and real-time surveillance, where resources are constrained but performance demands are high.
Innovation
This work introduces a parallel Mobile-Former architecture that innovatively employs:
1) A dual-branch design, with MobileNet for local features and a minimal-token Transformer for global context.
2) A lightweight bidirectional cross-attention mechanism that enables efficient feature exchange, reducing the need for dense attention.
3) Use of very few learnable tokens (e.g., 6) to encode global priors, significantly lowering computational overhead.
4) Integration of dynamic ReLU activations to improve non-linear modeling capacity.
These innovations collectively enable the model to maintain high accuracy with minimal FLOPs, addressing the efficiency-accuracy trade-off effectively.
Methodology
- �� Input images are processed by MobileNet-based blocks, capturing local details via depthwise and pointwise convolutions.
- �� At the bottleneck layer, a lightweight cross-attention module fuses local features with a small set of global tokens, which are randomly initialized and learned.
- �� The global tokens are processed by a Transformer module consisting of multi-head attention and feed-forward networks, with only a few tokens to keep costs low.
- �� Bidirectional cross-attention modules allow local features to inform global tokens (Mobile→Former) and vice versa (Former→Mobile), implemented with projection matrices selectively removed to save computation.
- �� Dynamic ReLU is employed in MobileNet blocks, with parameters generated from the first global token, enhancing non-linearity.
- �� The fused features are used for classification or detection, trained end-to-end with standard loss functions.
Experiments
The models are trained on ImageNet-1K using AdamW optimizer, 450 epochs, batch size 1024, with data augmentation techniques like Mixup and AutoAugment. Evaluation across FLOPs from 25M to 500M demonstrates consistent superiority over MobileNetV3 and ViT variants. For detection, Mobile-Former replaces backbone in RetinaNet and DETR, showing significant AP improvements and computational savings. Ablation studies analyze token number, dimension, and attention modules, confirming the design choices. The models are also tested on COCO for object detection, validating their transferability and robustness.
Results
Mobile-Former-294M achieves 77.9% top-1 accuracy, outperforming MobileNetV3 by 1.3%, with 17% fewer FLOPs. In RetinaNet detection, it gains 8.6 AP points over MobileNetV3. Using Mobile-Former as backbone in DETR, it improves AP by 1.1 points while reducing FLOPs by 52%. Ablation results show that reducing tokens below 6 degrades performance, but even a single token maintains high accuracy, demonstrating efficiency. The architecture consistently surpasses both CNNs and ViT variants across tasks.
Applications
The model is suitable for mobile and edge devices requiring real-time image classification and object detection. Its low FLOP count enables deployment in autonomous vehicles, surveillance cameras, and smartphones. The architecture also supports multi-task learning, making it adaptable for broader applications like video analysis and multi-modal systems. Its efficiency facilitates large-scale deployment in industry, enabling smarter, faster AI-powered solutions.
Limitations & Outlook
Despite its strengths, the model's performance diminishes in ultra-low FLOP regimes (<10M FLOPs). Its reliance on few global tokens may limit modeling capacity for highly complex scenes. Transferability to video or multi-modal tasks needs further validation. Additionally, optimizing token learning and attention mechanisms for hardware efficiency remains an ongoing challenge. Future work should address these issues to broaden applicability and robustness.
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有两个主要部门：一个负责制造零件（就像MobileNet），专注于细节和局部信息；另一个负责整体设计（像Transformer），需要了解整个产品的布局。以前，这两个部门合作很慢，因为他们要不停传递信息，效率很低。现在，工厂引入了一座轻巧的桥梁，让两个部门可以快速交流。制造部门把零件送到桥上，设计部门用少量的全局指示牌（少量Token）告诉制造部门整体的方向。反过来，设计部门也会根据制造部门的反馈调整设计。这样，两个部门可以同时工作，既快又高效，生产出更好的产品。这种设计让工厂既节省资源，又能应对复杂的订单，效率大大提高。
ELI14 Explained like you're 14
想象你在学校，有两个特别的团队：一个专门做作业（像MobileNet），他们很擅长处理细节；另一个负责制定大计划（像Transformer），需要知道整个班级的情况。以前，这两个团队合作很慢，因为信息传递太多太复杂。现在，有一座轻巧的桥，把他们连接起来，让他们可以快速交流。作业团队告诉大计划团队他们的细节，大计划用少量的全局指示牌告诉每个人整体方向。反过来，大计划也会把整体想法反馈给作业团队，让他们知道怎么调整。这样，两个团队合作得更快、更好，完成任务也更棒。这就像用少量的全局信息让每个人都知道大方向，又能关注细节，效率大大提高。这个方法让复杂的事情变得简单又高效，特别适合时间紧、资源少的场景。
Glossary
Cross Attention（交叉注意力）
一种机制，让两个不同的特征集相互关注，从而融合信息。技术上通过查询和键值机制实现，能在不同特征间建立联系。
在Mobile-Former中，用于局部特征与全局Token的双向信息交互。
Global Tokens（全局Token）
少量可学习的参数向量，用于编码图像的全局先验信息。技术上是随机初始化，通过训练学习代表全局特征。
在Transformer部分，用于捕获全局信息，减少计算成本。
Dynamic ReLU（动态ReLU）
一种激活函数，根据输入动态生成参数，增强模型非线性表达能力。技术上由MLP生成参数，适应不同输入。
在Mobile-Former中，用于提升局部特征的表达能力。
Parallel Architecture（并行架构）
将不同模块同时运行，彼此通过机制交互，而非串联。技术上提高效率，减少信息瓶颈。
Mobile-Former的核心设计思想。
Open Questions Unanswered questions from this research
- 1 如何在更低FLOPs（如10M）下保持性能，仍需研究。模型在极端低算力环境下表现有限，未来需优化全局特征编码策略。
- 2 模型在多模态、多任务场景中的迁移能力尚未充分验证，未来需扩展应用范围。
Abstract
We present Mobile-Former, a parallel design of MobileNet and transformer with a two-way bridge in between. This structure leverages the advantages of MobileNet at local processing and transformer at global interaction. And the bridge enables bidirectional fusion of local and global features. Different from recent works on vision transformer, the transformer in Mobile-Former contains very few tokens (e.g. 6 or fewer tokens) that are randomly initialized to learn global priors, resulting in low computational cost. Combining with the proposed light-weight cross attention to model the bridge, Mobile-Former is not only computationally efficient, but also has more representation power. It outperforms MobileNetV3 at low FLOP regime from 25M to 500M FLOPs on ImageNet classification. For instance, Mobile-Former achieves 77.9\% top-1 accuracy at 294M FLOPs, gaining 1.3\% over MobileNetV3 but saving 17\% of computations. When transferring to object detection, Mobile-Former outperforms MobileNetV3 by 8.6 AP in RetinaNet framework. Furthermore, we build an efficient end-to-end detector by replacing backbone, encoder and decoder in DETR with Mobile-Former, which outperforms DETR by 1.1 AP but saves 52\% of computational cost and 36\% of parameters.