Lightweight Vision Transformer Compression for On-Device Plant Disease Detection in Resource-Constrained Agricultural Field Conditions
Proposed H-BAC framework combines quantization and knowledge distillation, achieving 95.13% accuracy and 54.5x model compression.
Key Findings
Methodology
The paper presents a lightweight Vision Transformer compression framework combining Hessian-Balanced Adaptive Block Pruning (H-BAC), quantization, and attention-based knowledge distillation. Each technique is first evaluated independently through controlled ablation studies, after which the best-performing components are integrated into a sequential deployment pipeline tailored to real-world agricultural constraints.
Key Results
- On a chilli 3-class village-split dataset, compressed models match or exceed the 95.13% FP32 baseline's accuracy, alongside 74-98% model size reduction.
- The fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 ± 2.32% accuracy.
- A directly-trained student of the same final size reaches comparable accuracy of 94.87%, indicating where H-BAC and knowledge distillation are worth their computational cost.
Significance
The study is significant for academia and industry, addressing the long-standing challenge of deploying Vision Transformers on resource-constrained devices. By combining multiple compression techniques, it provides an efficient solution suitable for plant disease detection in agriculture, especially in rural India.
Technical Contribution
Technical contributions include the H-BAC pruning method, combining Hutchinson-estimated block-level Hessian curvature with first-order Taylor-based component pruning. An attention-based knowledge distillation scheme is introduced, addressing feature-dimension mismatch issues.
Novelty
This is the first joint evaluation of compression methods for agricultural disease detection, with H-BAC assigning non-uniform pruning rates based on second-order Hessian curvature, unlike traditional methods.
Limitations
- H-BAC and knowledge distillation's computational cost may not be justified in some scenarios, especially with directly trained student models.
- Pruning and quantization may lead to performance degradation on certain devices.
- Dataset diversity may not cover all possible agricultural scenarios.
Future Work
Future directions include further optimizing compression algorithms for efficiency, exploring more agricultural application scenarios, and developing more robust models to handle varying environmental conditions.
AI Executive Summary
Chilli is an economically significant crop in India, yet its productivity is threatened by diseases that are difficult to identify. While Vision Transformers have achieved high classification accuracy, their large computational footprint makes deployment on resource-constrained devices challenging. Existing compression approaches typically address pruning, quantization, and knowledge distillation in isolation, leaving the potential benefits and interactions of their combined application insufficiently explored. We propose a unified Vision Transformer compression framework that combines Hessian-Balanced Adaptive Block Pruning (H-BAC), guided by second-order sensitivity estimation, with quantization and attention-based knowledge distillation. To systematically identify the most effective configuration within each compression family, each technique is first evaluated independently through controlled ablation studies, after which the best-performing components are integrated into a sequential deployment pipeline tailored to real-world agricultural constraints. On a chilli 3-class village-split dataset with a genuine cross-village, cross-device out-of-distribution test split, the resulting compressed models match or exceed the 95.13% FP32 baseline's accuracy, alongside 74-98% model size reduction, and the fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 ± 2.32% accuracy across four tested configurations. A direct comparison further reveals that, on this dataset, a directly-trained student of the same final size, without pruning or distillation, reaches comparable accuracy of 94.87%, at the same 6.01 MB INT8 size, indicating where H-BAC and knowledge distillation are, and are not yet shown to be, worth their computational cost.
Deep Analysis
Background
Recent years have seen significant advances in automated plant disease detection technologies. Early methods relied on traditional machine learning and handcrafted features such as color histograms and texture descriptors. However, these methods had limited generalization capabilities under real field conditions. The advent of Convolutional Neural Networks (CNNs) transformed the scene, achieving 95% classification accuracy on the PlantVillage dataset. Vision Transformers (ViTs) further pushed performance, becoming the state of the art for plant disease classification. However, ViTs' computational cost is high, making them difficult to deploy on low-end mobile devices.
Core Problem
Chilli is an important economic crop in India, but its productivity is threatened by diseases. Traditional diagnostic methods rely on expert checks or laboratory testing, which are costly and slow, making them infeasible for large-scale application in rural India. An automated system is needed that can detect and identify plant diseases directly in the field, especially at the earliest stages.
Innovation
The paper proposes a lightweight Vision Transformer compression framework combining Hessian-Balanced Adaptive Block Pruning (H-BAC), quantization, and attention-based knowledge distillation. H-BAC assigns non-uniform pruning rates based on second-order Hessian curvature, differing from traditional methods. The attention-based knowledge distillation scheme addresses feature-dimension mismatch issues.
Methodology
- �� Use H-BAC for pruning, assigning pruning rates based on Hessian curvature.
- �� Apply quantization techniques to convert the model to INT8 format.
- �� Use attention-based knowledge distillation to transfer attention maps from the teacher model.
- �� Conduct ablation studies to evaluate the independent effect of each technique.
- �� Integrate the best components into a sequential deployment pipeline.
Experiments
Experiments use a chilli 3-class village-split dataset for cross-village and cross-device testing. The baseline model is ViT-B/16, pre-trained on ImageNet-1K. Evaluation metrics include accuracy, precision, recall, and F1 score. Ablation studies assess the independent effect of each compression technique.
Results
Compressed models on the chilli dataset match or exceed the 95.13% FP32 baseline's accuracy, achieving 74-98% model size reduction. The fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 ± 2.32% accuracy.
Applications
This technology can be used for plant disease detection on resource-constrained devices, especially suitable for rural India. Requires certain computational capabilities and camera equipment. Significant impact on agricultural productivity and disease management.
Limitations & Outlook
While compression techniques are effective, performance may degrade on certain devices. Dataset diversity may not cover all possible agricultural scenarios. Future work needs to further optimize algorithms for efficiency.
Plain Language Accessible to non-experts
Imagine a factory with many machines producing products. Each machine is important, but some machines may not need to be running all the time. Our research is like giving the factory a cleanup, turning off those machines that aren't used often, so the factory can operate more efficiently. We also provide new tools to the factory workers, allowing them to complete tasks faster. This is similar to how we compress and optimize the Vision Transformer model. By doing this, the factory not only saves resources but also improves production efficiency. This is the core idea of our research.
ELI14 Explained like you're 14
Imagine you're playing a super complex game with many levels and tasks. Every time you level up, you get new skills and gear that help you complete levels faster. Our research is like giving the game character an upgrade, allowing it to complete more tasks in less time. We also provide the character with new tools to better handle challenges in the game. This way, the game becomes not only more fun but also lets you win faster. That's the purpose of our research.
Glossary
Vision Transformer
A deep learning model based on attention mechanisms that can simultaneously process multiple regions of an image.
Used for plant disease classification, achieving high accuracy.
Model Compression
Methods to reduce model size and computational load, including pruning, quantization, and knowledge distillation.
Used to deploy Vision Transformers on resource-constrained devices.
Knowledge Distillation
A method to transfer knowledge from a large model to a smaller one, usually through soft labels.
Used to enhance the performance of compressed models.
Pruning
A method to reduce model complexity by removing unimportant parameters.
Used in H-BAC to reduce the computational load of Vision Transformers.
Quantization
A method to convert model parameters to a lower precision format to reduce computational load.
Used to convert the model to INT8 format.
Open Questions Unanswered questions from this research
- 1 How to further optimize compression algorithms for efficiency and performance?
- 2 How to improve model robustness under varying environmental conditions?
- 3 How to expand the dataset to cover more agricultural scenarios?
Applications
Immediate Applications
Agricultural Disease Detection
Can be used for real-time detection and identification of plant diseases, helping farmers improve productivity.
Deployment on Resource-Constrained Devices
Suitable for low-end mobile devices, providing efficient disease detection solutions.
Long-term Vision
Smart Agriculture Management
Improves agricultural productivity through automated disease detection, advancing agricultural technology.
Abstract
Chilli (Capsicum annuum) is one of India's most economically significant crops, yet its productivity is persistently threatened by diseases that are difficult to identify without expert intervention. While Vision Transformers (ViTs) have achieved high classification accuracy, their large computational footprint makes deployment on resource constrained devices challenging. Existing compression approaches typically address pruning, quantization, and knowledge distillation in isolation, leaving the potential benefits and interactions of their combined application insufficiently explored. We propose a unified Vision Transformer compression framework that combines Hessian-Balanced Adaptive Block Pruning (H-BAC), guided by second-order sensitivity estimation, with quantization and attention-based knowledge distillation. To systematically identify the most effective configuration within each compression family, each technique is first evaluated independently through controlled ablation studies, after which the best-performing components are integrated into a sequential deployment pipeline tailored to real-world agricultural constraints. On a chilli 3-class village-split dataset with a genuine cross-village, cross-device out-of-distribution test split, the resulting compressed models match or exceed the 95.13% FP32 baseline's accuracy, alongside 74-98% model size reduction, and the fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 +/- 2.32% accuracy across four tested configurations. A direct comparison further reveals that, on this dataset, a directly-trained student of the same final size, without pruning or distillation, reaches comparable accuracy of 94.87%, at the same 6.01 MB INT8 size, indicating where H-BAC and knowledge distillation are, and are not yet shown to be, worth their computational cost.