FTerViT: Fully Ternary Vision Transformer
FTerViT fully ternarizes all weights and normalization parameters, achieving 82.43% accuracy with 15× compression on ImageNet.
Key Findings
Methodology
FTerViT introduces fully ternarized components: TernaryBitConv2d for patch embedding and TernaryLayerNorm for normalization layers. The training employs knowledge distillation from a full-precision teacher, combined with a two-phase quantization-aware fine-tuning strategy. The approach uses per-channel scaling for convolutional weights and normalization parameters, with straight-through estimators guiding gradient flow. The model is trained on ImageNet-1K, utilizing cosine decay learning rate schedules, and validated across multiple resolutions. The key innovation lies in extending ternarization beyond encoder layers to include all sensitive components, ensuring minimal accuracy loss.
Key Results
- At 384×384 resolution, the W2A8 DeiT-III-S model reaches 82.43% top-1 accuracy on ImageNet-1K, with a model size of 6.09MB, representing a 15× compression and outperforming previous ternary ViT methods by up to 8 percentage points.
- At 224×224 resolution, the DeiT-III-Small-based FTerViT achieves 79.64% accuracy with a model size of 5.81MB, deployable on ESP32-S3 microcontroller, demonstrating practical edge deployment.
- The fully ternarized model significantly reduces storage, energy consumption by 55%, and inference latency, validating its suitability for resource-constrained edge devices, with real-world experiments confirming feasibility.
Significance
This work pushes the boundary of model compression by enabling a fully ternary Vision Transformer, addressing critical bottlenecks in deploying large models on microcontrollers. It demonstrates that all components, including traditionally sensitive layers like patch embedding and LayerNorm, can be quantized to ternary without substantial accuracy degradation. The methodology combines innovative operator design and training strategies, paving the way for ultra-lightweight yet high-performance models suitable for real-time, low-power applications. The successful hardware implementation on ESP32-S3 underscores its practical relevance, opening new avenues for edge AI deployment in IoT, robotics, and mobile devices.
Technical Contribution
The paper introduces TernaryBitConv2d and TernaryLayerNorm operators, extending ternarization to all model parameters. It develops a two-phase knowledge distillation and fine-tuning pipeline that stabilizes training and preserves accuracy. The approach leverages per-channel scaling and importance-based sensitivity analysis to identify and quantize all critical layers, including patch embedding and normalization. The resulting models achieve state-of-the-art accuracy among fully ternarized ViTs, with significant compression ratios. Hardware implementation on ESP32-S3 validates the method's practicality, enabling fully on-device inference with minimal resource footprint.
Novelty
This is the first work to fully ternarize all parameters of a Vision Transformer, including patch embedding and LayerNorm, which were previously considered too sensitive. The design of TernaryBitConv2d and TernaryLayerNorm operators, combined with a two-stage knowledge distillation process, represents a significant advancement over prior partial quantization methods. The ability to deploy high-accuracy models on microcontrollers with only a few megabytes of memory sets a new standard in edge AI, bridging the gap between model size and performance in resource-limited environments.
Limitations
- Despite minimal accuracy loss, the model still exhibits a small performance gap (~2.4%) compared to full-precision counterparts, which may be critical in some applications requiring utmost precision.
- Current implementation is tailored to DeiT-III architectures; extending to larger or different transformer variants may require additional tuning.
- Inference speed, while improved, is still limited by hardware constraints; further optimization is needed for real-time applications on ultra-low-power devices.
Future Work
Future research will explore adaptive training strategies to further close the accuracy gap, investigate hardware-aware quantization techniques, and extend the approach to other vision architectures. Additionally, integrating neural architecture search could optimize model structures for specific edge hardware, and multi-task learning could broaden application scope. The ultimate goal is to develop universally deployable, ultra-lightweight transformers that maintain high accuracy across diverse tasks and devices.
AI Executive Summary
Transformers have revolutionized visual recognition, but their large size hampers deployment on resource-constrained devices like microcontrollers. Traditional models such as DeiT-Small demand over 80MB in FP32, far exceeding typical microcontroller memory. To address this, the authors propose FTerViT, a fully ternarized Vision Transformer that compresses all parameters—including patch embedding, normalization layers, and classifier head—to just a few megabytes without significant accuracy loss.
The core innovation involves designing new operators: TernaryBitConv2d for patch embedding and TernaryLayerNorm for normalization, both capable of handling the sensitivity of these layers during quantization. The training employs a two-stage knowledge distillation process, first stabilizing the model with a higher learning rate, then fine-tuning with a lower rate to recover performance. Extensive experiments on ImageNet-1K demonstrate that the 384×384 model achieves 82.43% top-1 accuracy at only 6.09MB, surpassing previous ternary approaches by up to 8 percentage points. The 224×224 version based on DeiT-III-Small reaches 79.64% accuracy, deployable on ESP32-S3 microcontrollers.
Hardware validation confirms that the fully ternary model runs efficiently on a dual-core microcontroller, reducing energy consumption by 55% and inference latency significantly. This work marks a breakthrough in ultra-lightweight vision models, enabling high-performance AI on tiny, low-power devices. Future directions include optimizing training pipelines, expanding to other architectures, and integrating hardware-aware quantization to further enhance edge AI capabilities.
Deep Analysis
Background
近年来，Transformer架构在视觉识别中表现出色，代表作如ViT、DeiT不断推动模型性能提升。然而，庞大的参数规模带来存储和计算的巨大挑战，限制了其在微控制器等边缘设备上的应用。传统量化技术如INT8、INT4虽然减小了模型体积，但敏感层如patch embedding和LayerNorm仍需高精度参数，导致整体模型依然庞大。近年来，极端压缩技术如二值化、三值化逐渐兴起，但多局限于encoder层，未能全面覆盖所有参数。本文在此基础上，提出全参数三值化方案，旨在突破边缘设备部署瓶颈。
Core Problem
核心问题在于如何在极端压缩条件下，保持Transformer模型的识别性能。尤其是patch embedding、LayerNorm和分类头等敏感层，传统量化会引起较大性能下降。现有研究多选择保留这些层的高精度参数，导致模型体积仍然偏大，难以在存储空间有限的微控制器上部署。如何设计适合全参数三值化的算子，确保模型在极端压缩下的鲁棒性，是亟待解决的难题。
Innovation
本研究的创新点在于：1）提出全三值化架构，覆盖所有关键参数，包括patch embedding和LayerNorm；2）引入TernaryBitConv2d和TernaryLayerNorm两个新算子，解决敏感层的量化难题；3）采用两阶段知识蒸馏训练策略，确保模型性能的稳定提升。4）在硬件实现方面，首次在ESP32-S3微控制器上实现全三值化推理，验证其实际应用潜力。这些创新共同推动了极端模型压缩技术的发展，突破了传统只对encoder层进行三值化的限制。
Methodology
- �� 设计全三值化的算子：TernaryBitConv2d用于patch embedding，采用per-channel缩放；TernaryLayerNorm对LayerNorm参数进行三值化，保持归一化效果。• 采用知识蒸馏：从预训练FP32教师模型中蒸馏知识，提升学生模型的表现。• 两阶段训练：第一阶段以较大学习率训练至收敛，第二阶段低学习率微调，恢复性能。• 量化感知微调：结合梯度直通估计，优化三值参数。• 训练过程中，利用余弦学习率调度，确保模型稳定收敛。• 采用Taylor FO重要性分析，识别敏感层，指导量化策略。• 最终模型在ImageNet-1K上验证，达到82.43%的准确率，模型大小仅为6.09MB。
Experiments
在ImageNet-1K数据集上，采用DeiT-III-S和DeiT-III-Small作为基线，比较不同量化方案的性能。设置两阶段训练策略，超参数包括学习率1e-4和1e-5，训练260轮。通过Taylor FO和Hessian-trace分析，验证patch embedding和LayerNorm的敏感性。对比不同模型尺寸和分辨率的性能，验证全三值化的鲁棒性和压缩效果。还在ESP32-S3微控制器上实现端到端推理，测试能耗和速度。
Results
模型在384×384分辨率下，达成82.43%的ImageNet-1K top-1准确率，模型大小6.09MB，压缩比达15倍，优于之前的三值化方法最多8个百分点。在224×224分辨率下，基于DeiT-III-Small的模型实现79.64%的准确率，模型在微控制器上实现全自主推理，验证了其边缘应用潜力。模型在存储、能耗和推理速度方面表现优异，能耗降低55%，推理时间缩短至21秒，验证了极端压缩的实用性。
Applications
该全三值化模型适用于边缘设备的实时视觉识别任务，如智能监控、无人机导航、智能家居等。其低存储需求和低能耗特性，使其能在微控制器、IoT设备上部署，满足低功耗、低延迟的应用需求。未来可结合硬件感知的量化方案，推动端侧AI在工业、安防、医疗等行业的普及。
Limitations & Outlook
模型在极端压缩条件下仍存在少量性能损失（约2.4个百分点），尤其在更大规模模型或更复杂任务中，性能可能进一步下降。当前实现主要针对特定架构（DeiT-III系列），迁移到其他Transformer变体或任务仍需调优和验证。推理速度受限于硬件平台，尽管在ESP32-S3上实现了部署，但在更低端微控制器上可能面临性能瓶颈。
Plain Language Accessible to non-experts
想象你有一个超级复杂的工厂，里面有许多机器在生产各种商品。每台机器都很重要，但有些机器非常敏感，一点点调整就会影响整个生产线。以前，为了让工厂更小、更快，工程师们会把一些机器的设置调得很精细，甚至用特殊的方式让它们变得更简单，比如用只有三种状态的开关（开、关、空闲）来代替复杂的调节。这就像把工厂里的每个机器都变成了只会开、关或空闲的三态开关。这样，工厂的整体体积变小了，运行也更快了，但如果不小心，可能会影响生产效率。现在，这个工厂的工程师们找到了一套聪明的方法，让所有机器都用三态开关，而且还能保持生产效率。这就像把所有机器都变成了只会开、关或空闲的开关，但通过巧妙的调节和学习，工厂仍然能高效运转。这种方法让工厂变得更小、更快，也更节能，未来还能用在各种不同的工厂里，帮它们变得更智能、更高效。
ELI14 Explained like you're 14
想象你在学校里有很多不同的学习工具，比如笔、尺子、计算器。以前，这些工具都很复杂，有很多按钮和功能，但用起来很麻烦。现在，想象你用的工具变得超级简单，只剩下三种状态：开、关和休眠。这样，你的书包就变得更轻了，拿起来也更快了，但你还是能完成所有任务。科学家们做的事情就像这样：他们让一个超级聪明的电脑模型变得更简单，只用三种状态来表示所有的数字和参数。虽然变得简单了，但模型依然能像以前一样聪明，甚至更快、更省电。这就像用一个简单的开关控制一台复杂的机器，但这个开关经过特别设计，能让机器保持原有的功能。这样，未来我们就可以把这些超级聪明的模型放在小小的手机或微控制器里，让它们帮我们做事情，比如识别图片、翻译语言，甚至控制家里的智能设备。是不是很酷？科学家们用这种方法，让复杂的模型变得更简单、更快，也更节能，未来的智能设备会变得更小、更强大！
Abstract
Ternary Vision Transformers offer substantial model compression, however state-of-the-art methods only ternarize the encoder layers, leaving patch embeddings, LayerNorm parameters, and classifier heads in full precision. In compact models targeting resource-constrained processors, such as microcontrollers, these remaining full-precision components determine the total memory footprint, severely limiting deployment efficiency and on-device feasibility. In this work, we introduce a fully ternarized Vision Transformer in which \emph{all} weight matrices and normalization parameters are ternarized (FTerViT). To this end, we introduce two novel operators : TernaryBitConv2d with per-channel scaling for patch embedding and TernaryLayerNorm. FTerViT is trained using knowledge distillation, followed by a lightweight quantization-aware recovery phase. Our ternary W2A8 DeiT-III-S at 384$\times$384 resolution achieves 82.43\% ImageNet-1K top-1 at 6.09\,MB (${\sim}$15$\times$ compression, $-$2.42\,pp vs.\ FP32), outperforming prior ternary ViTs methods up to 8 pp. Finally, we demonstrate the first implementation of ternary vision transformers on a dual cores XTensa LX7 microcontroller inside the ESP32-S3 system-on-chip. By deploying FTerViT-Small (based on DeiT-III-Small at 224$\times$224 resolution, 5.81\,MB), we achieve 79.64\% ImageNet-1K top-1 accuracy.