A Dual-Path Architecture for Scaling Compute and Capacity in LLMs

Looped transformers apply a shared block multiple times, emerging as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs, looped models have strictly less capacity than baseline transformers. This paper proposes a novel dual-path block architecture that can flexibly scale compute and capacity. It exposes both axes as parallel pathways within a single layer: a deep sublayer re-applied K times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once. Independent per-token gates combine both axes and allow detailed per-token routing analyses. Experimental results show that across two FLOP budgets, the dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations, while using fewer parameters than the baseline. Learned gates are directly interpretable and show systematic per-token allocation: function words trend wide, while punctuation, symbols, and arithmetic tokens trend deep. This research addresses the capacity limitation of looped transformers at fixed FLOPs by proposing a novel dual-path architecture, offering flexible compute and capacity scaling, significant for academia and industry. Future directions include exploring the application of dual-path architecture in other models and optimizing gating mechanisms for better performance.