Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis
By orchestrating small language models, the hybrid system of Qwen3-4B and Foundation-Sec-8B outperforms single LLMs in malware analysis with 35.30% accuracy.
Key Findings
Methodology
The study employs four architectures: multi-agent pipeline, adversarial debate framework, hierarchical consultation system, and hybrid architecture. Each architecture enhances the capabilities of small language models in answering structured questions about malware detonation reports.
Key Results
- The hybrid system achieved 35.30% overall accuracy on Meta's CyberSecEval benchmark, surpassing the strongest cyber-specialized baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%).
- With the same evidence pipeline, the evidence-grounded Gemini configuration achieved 38.22% accuracy, demonstrating that evidence-driven orchestration can significantly improve collaborative SLM performance.
- Case studies show the hybrid system can correct reasoning errors on novel evasion techniques like EtherHiding and ClickFix.
Significance
The study demonstrates that evidence-driven orchestration allows small language models to provide performance comparable to large language models in malware analysis. This offers viable solutions for resource-constrained environments, reducing computational resources and hosting costs.
Technical Contribution
The study introduces four novel orchestration architectures, showcasing how task decomposition, adversarial debate, and expert consultation can enhance small language model performance. These architectures provide new theoretical guarantees and engineering possibilities in malware analysis.
Novelty
This is the first application of orchestrated small language models in malware analysis, demonstrating how evidence-driven orchestration can outperform single large language models in complex reasoning tasks.
Limitations
- The hybrid system may introduce unnecessary information when handling simple questions, affecting accuracy.
- Further research is needed to optimize the synergy between evidence collection and debate phases.
Future Work
Future research can explore how to further optimize the performance of orchestration architectures and apply them to structured problem-solving in other domains.
AI Executive Summary
Malware analysis is a critical bottleneck in cybersecurity, requiring rapid interpretation of complex detonation reports. While large language models excel at interpreting technical artifacts, their closed nature and high API costs motivate exploration of open-weight alternatives. This paper investigates whether orchestrated ensembles of small language models can match or exceed single LLM performance on structured questions about malware detonation reports. We designed and evaluated four orchestration architectures, including multi-agent pipeline, adversarial debate framework, hierarchical consultation system, and hybrid architecture. The hybrid system achieved 35.30% overall accuracy on Meta's CyberSecEval benchmark, surpassing the strongest cyber-specialized baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%). These findings show that evidence-driven orchestration can substantially improve the performance of collaborative SLMs, offering viable solutions for resource-constrained environments. Future research can explore how to further optimize the performance of orchestration architectures and apply them to structured problem-solving in other domains.
Deep Analysis
Background
Malware analysis remains a critical bottleneck in cybersecurity operations. Analysts must rapidly triage suspicious samples, interpret complex detonation reports, and assess threat severity under time pressure. Traditional static and dynamic analysis pipelines generate rich telemetry, but extracting actionable intelligence from these multi-faceted reports demands expert knowledge of malware techniques and attack frameworks.
Core Problem
Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviors. While large language models demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives.
Innovation
The paper introduces four novel orchestration architectures: multi-agent pipeline, adversarial debate framework, hierarchical consultation system, and hybrid architecture. These architectures enhance small language model performance through task decomposition, adversarial debate, and expert consultation.
Methodology
- �� Multi-agent pipeline: decomposes analysis into structured evidence-collection and reasoning stages. • Adversarial debate framework: two agents iteratively critique each other's reasoning. • Hierarchical consultation system: pairs a general-purpose SLM with a cyber-specialized expert model. • Hybrid architecture: combines evidence-grounded pipelines with adversarial debate reasoning.
Experiments
We tested eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest baselines.
Results
The hybrid system achieved 35.30% overall accuracy on Meta's CyberSecEval benchmark, surpassing the strongest cyber-specialized baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%).
Applications
The findings can be applied in resource-constrained environments, providing efficient solutions for malware analysis, reducing computational resources and hosting costs.
Limitations & Outlook
The hybrid system may introduce unnecessary information when handling simple questions, affecting accuracy. Further research is needed to optimize the synergy between evidence collection and debate phases.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Large language models are like experienced chefs who can quickly make delicious dishes but require expensive ingredients and equipment. Small language models are like novice chefs who, while not as skilled, can still make tasty dishes through collaboration and using simple tools. This paper explores how coordinating multiple novice chefs can outperform a single experienced chef.
ELI14 Explained like you're 14
Imagine you're playing a multiplayer game. Large language models are like super players who can quickly complete tasks but need expensive gear. Small language models are like regular players who, while not as equipped, can complete tasks through teamwork and strategy. This paper explores how coordinating multiple regular players can outperform a single super player.
Glossary
SLM (Small Language Model)
Refers to language models with fewer parameters that can run on consumer hardware.
Used for structured problem-solving in malware analysis.
LLM (Large Language Model)
Refers to language models with more parameters requiring high-performance computing resources.
Used for technical artifact interpretation.
CyberSecEval
Malware analysis benchmark provided by Meta.
Used to evaluate language model capabilities on security tasks.
Malware Detonation Report
Complex reports containing filesystem, network, and process behaviors.
Used for malware analysis.
Adversarial Debate Framework
Two agents iteratively critique each other's reasoning.
Used to enhance small language model reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 How to further optimize orchestration architecture performance?
- 2 How to apply these architectures to structured problem-solving in other domains?
Applications
Immediate Applications
Resource-Constrained Environments
Provides efficient solutions for malware analysis, reducing computational resources and hosting costs.
Long-term Vision
Broad Application
Apply orchestration architectures to structured problem-solving in other domains, advancing automated analysis.
Abstract
Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.