A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

The rapid evolution of neural language models (LMs) has transformed NLP, with architectures like BERT, T5, and GPT-3 pushing performance boundaries. These models leverage massive parameters—up to hundreds of billions—driven by the neural scaling law, which correlates size with emergent abilities. However, such large models require extensive computational resources, high energy consumption, and pose privacy risks, limiting their deployment in real-world, resource-constrained scenarios. Consequently, research shifted toward developing Small Language Models (SLMs), which aim to retain key capabilities while reducing size and complexity. Techniques like pruning, knowledge distillation, and quantization have been extensively explored to optimize models for edge devices, mobile applications, and domain-specific tasks. Despite progress, defining and standardizing SLMs remains challenging, with diverse criteria based on parameter count, task performance, and resource constraints. This survey consolidates recent advances, providing a unified framework to guide future development.

Large models, though powerful, face practical limitations: high computational costs, latency, privacy concerns, and difficulty in deployment on edge devices. Existing definitions based solely on parameter count are inconsistent, leading to ambiguity in standardization. Additionally, models often underperform in specialized domains like healthcare and law due to insufficient domain-specific knowledge. Fine-tuning large models is computationally expensive, hindering rapid adaptation. These issues necessitate a clear, task-oriented, resource-aware definition of SLMs, along with effective techniques to optimize their performance within resource constraints, ensuring broader accessibility and application.

This work introduces a capability-resource boundary framework for SLMs, emphasizing task-specific performance thresholds aligned with resource limitations. It systematically categorizes optimization techniques—pruning, distillation, quantization—and evaluates their effectiveness across datasets. The novelty lies in moving beyond parameter count as the sole metric, focusing instead on functional capabilities and resource feasibility. The framework facilitates standardized evaluation, guiding model design tailored for edge deployment and domain-specific applications. By integrating empirical performance data, the approach offers a practical pathway for developing efficient, reliable, and adaptable SLMs that meet real-world demands.

• �� Analyze existing Transformer architectures, focusing on self-attention and feedforward modules.
• �� Implement pruning strategies—structured and unstructured—to remove redundant parameters.
• �� Apply knowledge distillation, transferring knowledge from large models (e.g., GPT-3) to smaller ones.
• �� Use quantization-aware training to reduce precision, lowering memory and computation.
• �� Develop multi-task evaluation metrics, combining accuracy, latency, and energy consumption.
• �� Conduct experiments on datasets like GLUE, SQuAD, and domain-specific corpora, comparing models of varying sizes.
• �� Optimize models for edge deployment by balancing size, speed, and accuracy, validating on hardware like smartphones and embedded systems.

Models ranging from 100 million to 10 billion parameters were trained and evaluated across multiple datasets. Baselines included large models like GPT-3 and Llama-3.1. Performance metrics included accuracy, F1, inference latency, and energy consumption. Techniques such as pruning and distillation were systematically applied, with ablation studies to assess their impact. Experiments demonstrated that models with 100M–1B parameters could achieve 80-90% of large model performance on tasks like GLUE, with inference times reduced by over 50%. Edge deployment tests confirmed the feasibility of real-time operation within resource budgets. These results validate the effectiveness of the proposed definition and optimization strategies.

SLMs with 100 million to 1 billion parameters achieved an average of 85.3 on GLUE, close to larger models. Knowledge distillation reduced model size by 50% with only 3% performance drop. In medical and legal tasks, lightweight models improved inference speed by over 2x, meeting real-time requirements. Quantization further decreased energy consumption by 30%, confirming suitability for edge devices. These findings demonstrate that carefully optimized small models can match or surpass large models in specific applications, validating the resource-capability framework.

SLMs are ideal for mobile assistants, IoT devices, and privacy-sensitive sectors like healthcare and finance. They enable real-time, local inference, reducing reliance on cloud services. Customized lightweight models facilitate rapid deployment in industry-specific scenarios, such as medical diagnostics, legal document analysis, and personalized content filtering. Their low latency and energy efficiency support broader AI adoption in resource-limited environments, fostering democratization of AI technology.

Despite promising results, SLMs still face challenges in robustness, generalization, and security. Extreme resource constraints can lead to performance degradation, especially in complex tasks. Model compression techniques may introduce biases or hallucinations, affecting trustworthiness. Hardware-specific optimization remains complex, and multi-modal integration is underexplored. Future work must address these issues, focusing on robustness, explainability, and safety to ensure reliable deployment across diverse real-world scenarios.