Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference
Daedalus-150M employs a convolution-attention hybrid with 6 full attention and 12 convolution layers, trained on 59.9B tokens, outperforming similar models in quality and inference speed.
Key Findings
Methodology
This work adopts a reverse design approach, starting from hardware constraints—single user, single token, 4-bit weights, ordinary CPU—to determine architecture. The model consists of 18 layers, with 6 full attention layers interleaved with 12 fixed-width short convolution layers. Convolution states are fixed at two timesteps, avoiding repeated cache reads during long conversations. Training used AdamW and Muon optimizers on 599 billion tokens, with linear learning rate decay. The model was evaluated against a parameter-matched all-attention baseline, demonstrating superior inference speed and comparable or better quality.
Key Results
- The model scored 47.31 on a five-task benchmark, surpassing the fixed target of 42.20, outperforming models trained on 3-6 times more data like GPT-2 124M, Pythia 160M, OPT-125M, and GPT-Neo 125M. Validation bits-per-byte was 0.8685.
- Compared to a parameter-matched all-attention model, the hybrid outperformed it by 0.81% on quality, produced a 6.3% smaller 4-bit file, and decoded 1.76 times faster at 2048 tokens of context, with a 2.08× speedup against a similar external model. The speed advantage increased with longer contexts, consistent with the mechanism's prediction.
- Pure 4-bit quantization introduced quality degradation, with roughly half of convolution channels becoming inert and difficult to prune; vocabulary size also impacted model compactness.
Significance
This research addresses the challenge of deploying high-quality language models efficiently on CPU hardware. By designing a hybrid architecture tailored to hardware constraints, it achieves significant improvements in inference speed and storage efficiency, enabling practical deployment on edge devices. The approach demonstrates that careful architecture choices, guided by hardware considerations, can surpass traditional models trained on more data, opening new pathways for resource-constrained NLP applications. It advances the understanding of how to balance attention and convolution layers for optimal performance in low-resource environments, influencing future model design and hardware-software co-optimization.
Technical Contribution
The core innovation lies in combining 6 attention layers with 12 convolution layers, leveraging fixed-width convolution states to eliminate the linear growth of cache reads during decoding. The use of grouped-query attention (GQA) reduces cache size further, and tying input-output embeddings cuts parameters. The model is trained with post-training quantization to 4 bits, optimizing storage and inference speed. The architecture’s design is explicitly aligned with CPU hardware constraints, validated through extensive ablation and comparison experiments, demonstrating superior speed and comparable quality at a fraction of the storage cost.
Novelty
This is the first systematic proposal of a CPU-optimized convolution-attention hybrid architecture based on reverse engineering from hardware constraints. Unlike existing models that rely heavily on full attention or pure convolution, this work strategically fixes convolution states and limits attention layers, balancing long-range retrieval with efficiency. The parameter-matched comparison confirms the architecture’s advantage, emphasizing the importance of hardware-aware design in NLP model development.
Limitations
- The model did not incorporate quantization-aware training (QAT), leading to performance degradation post-quantization. Future work could integrate QAT for better accuracy.
- Vocabulary size remains large at 49,152, increasing embedding parameters and storage, which could be reduced with more optimal tokenization.
- The architecture heavily depends on specific CPU hardware features; adapting it to other hardware platforms may require significant tuning.
Future Work
Future directions include integrating quantization-aware training to improve post-quantization performance, optimizing vocabulary size, and extending the architecture for multi-task and multi-user scenarios. Additionally, exploring hardware-specific accelerators and co-design strategies could further enhance deployment efficiency, broadening the applicability of such models in real-world edge devices and low-power environments.
AI Executive Summary
The rapid growth of large-scale language models has revolutionized NLP, yet deploying these models on resource-constrained devices remains a challenge. Traditional architectures, primarily designed for GPU acceleration, struggle with the limited memory bandwidth and compute capabilities of CPUs, especially in single-user, real-time scenarios. Addressing this gap, the Daedalus-150M introduces a novel convolution-attention hybrid architecture tailored specifically for CPU inference, emphasizing hardware-aware design principles.
This model strategically combines six full attention layers with twelve fixed-width short convolution layers, where convolution states are maintained at just two timesteps, ensuring constant memory access costs regardless of conversation length. The architecture was derived through reverse engineering, starting from hardware constraints—single token, 4-bit quantization, and typical CPU bandwidth—rather than adapting from GPU-centric models. Training on 599 billion tokens, the model achieves a five-task benchmark score of 47.31, surpassing fixed benchmarks and outperforming models trained on significantly more data.
Extensive experiments demonstrate that the hybrid architecture outperforms a parameter-matched all-attention baseline by 0.81% in quality, while decoding at 2048 tokens is 1.76 times faster. The model’s file size is also 6.3% smaller, illustrating efficiency gains. Notably, the speed advantage grows with longer contexts, aligning with the theoretical predictions of the mechanism. The study highlights that optimizing for hardware constraints can lead to architectures that outperform traditional models in both speed and storage, especially in edge deployment scenarios.
However, the work also reveals limitations, such as the absence of quantization-aware training and the large vocabulary size, which could be addressed in future iterations. Overall, this research opens new avenues for deploying high-performance language models on low-resource hardware, emphasizing the importance of hardware-aware architecture design and paving the way for more accessible NLP solutions in edge environments.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，传统的工厂用高速机器不停地读取大量原料，虽然效率高，但耗能大、占空间。而这项研究设计了一种新型工厂，使用少量的原料和简单的机器，通过巧妙的流程安排，让工厂既能快速完成订单，又节省能源和空间。它用一种聪明的方式，把复杂的任务拆成两个部分：一部分用高速的注意力机制处理长距离信息，另一部分用简单的卷积快速处理短距离信息。这样，工厂在处理每个订单时，不用不停地读取大量原料，节省了时间和能源。就像用一套聪明的流程，让工厂变得更高效、更节能，特别适合在资源有限的地方，比如手机或边缘设备。这种设计让电脑也能像工厂一样，既快又省钱，能在有限的条件下做出很棒的表现。
ELI14 Explained like you're 14
想象你在学校里，有两种学习方法：一种是每天花很多时间记忆所有细节，另一种是用聪明的技巧快速理解重点。这篇文章就像在告诉你，科学家们设计了一种特别的学习方法，让电脑像用聪明技巧一样，既能学得快，又能省电。它用了一些特别的“记忆袋”和“快速理解”的方法，把复杂的任务变得简单。比如，电脑不用每次都重新看全部的资料，而是用一种“记忆袋”存一些重要信息，只在需要的时候快速取出。这样，电脑就可以更快地回答问题，也不需要太多存储空间。这就像你用一个小背包，装着最重要的书和笔，走到哪里都能快速找到需要的东西。
Glossary
Transformer（变换器）
一种深度学习模型架构，利用注意力机制处理序列数据，广泛应用于自然语言处理。技术上，它通过多头注意力和位置编码实现长距离依赖。
本文中的模型架构核心，采用多层Transformer进行信息处理。
量化（Quantization）
将模型参数从高精度（如FP32）压缩到低比特（如4-bit），以减少存储和计算成本。技术上，通过量化算法保持模型性能。
本文采用后量化到4-bit，提升模型在CPU上的推理效率。
GQA（Grouped-Query Attention）
一种优化注意力机制的方法，通过分组查询减少缓存需求和参数量，提升推理速度。
模型中应用GQA以减轻注意力层的存储压力。
深度可分卷积（Depthwise Convolution）
一种卷积操作，将每个通道单独卷积，参数少、计算快，常用于轻量级模型。
短卷积层采用深度可分卷积，固定宽度为2，保持低延迟。
参数匹配（Parameter Matching）
在不同模型间保持参数数量一致，用于公平比较架构优劣。
对比全注意力模型验证混合架构的性能优势。
Open Questions Unanswered questions from this research
- 1 如何进一步优化混合架构在多任务、多模态场景下的表现仍未解决，尤其是在多用户环境中模型的适应性和扩展性有待研究。
- 2 量化感知训练（QAT）未在本研究中应用，未来需验证其在此架构中的性能提升潜力。
- 3 硬件迁移问题：该架构高度依赖特定硬件特性；在不同硬件平台上的适应性仍需探索。
Applications
Immediate Applications
边缘设备自然语言理解
利用该模型在智能手机、嵌入式设备上实现高效对话和问答，降低硬件成本，提升响应速度。
低功耗智能助手
在资源有限的场景中部署，支持实时语音识别和交互，改善用户体验。
Long-term Vision
普及高效AI硬件平台
推动硬件厂商采用定制化架构，将混合模型集成到各种边缘设备中，实现广泛应用。
Abstract
Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.