StrataCL: Fabric-Native Communication Library for Production Supernodes
StrataCL enables user-buffer direct communication via registration-on-allocation, boosting bandwidth by 1.6x and inference throughput by 1.9x.
Key Findings
Methodology
StrataCL introduces registration-on-allocation to eliminate redundant data copies. It utilizes workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to leverage supernode architecture features.
Key Results
- On Huawei CloudMatrix384, StrataCL increases collective bus bandwidth by 1.6x and MoE dispatch/combine bus bandwidth by 1.4x.
- Across three production workloads, StrataCL improves LLM inference throughput by 1.9x and reduces P99 TTFT by 2.2x.
- LLM and Recsys training iteration times are reduced by 1.4x and 1.3x, respectively.
Significance
StrataCL significantly enhances communication efficiency for distributed AI workloads by eliminating redundant data copies and optimizing communication operators. This advancement is crucial for both academia and industry, addressing long-standing communication bottlenecks.
Technical Contribution
StrataCL's technical contributions include a zero-redundancy communication path design and full utilization of supernode architecture features. Its registration-on-allocation mechanism and NPU-driven SDMA offloading offer new engineering possibilities for existing communication libraries.
Novelty
StrataCL is the first to achieve user-buffer direct communication in supernode architectures, significantly reducing communication latency. Its innovation lies in eliminating registration overhead and optimizing communication operators compared to existing methods.
Limitations
- Scalability of registration-on-allocation in large-scale scenarios needs improvement.
- Adaptability to different architectures requires further validation.
Future Work
Future work includes further optimizing the scalability of the registration-on-allocation mechanism and validating its applicability to other supernode architectures.
AI Executive Summary
Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries are largely buffer-centric, leading to redundant data copies or costly user-buffer registration. StrataCL introduces registration-on-allocation to enable user-buffer direct communication and designs workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to leverage supernode architecture features.
On Huawei CloudMatrix384, StrataCL improves collective bus bandwidth by up to 1.6x and MoE dispatch/combine bus bandwidth by up to 1.4x. Across three production workloads, StrataCL enhances LLM inference throughput by 1.9x and reduces P99 TTFT by 2.2x. LLM and Recsys training iteration times are reduced by 1.4x and 1.3x, respectively.
StrataCL's technical contributions include a zero-redundancy communication path design and full utilization of supernode architecture features. Its registration-on-allocation mechanism and NPU-driven SDMA offloading offer new engineering possibilities for existing communication libraries. Future work includes further optimizing the scalability of the registration-on-allocation mechanism and validating its applicability to other supernode architectures.
Deep Analysis
Background
As AI model sizes and cluster scales grow, distributed execution becomes the norm, making communication efficiency a key bottleneck. Existing communication libraries like NCCL, RCCL, etc., rely heavily on buffer management, leading to redundant data copies. Supernode architectures like Huawei CloudMatrix384 offer new optimization opportunities.
Core Problem
Existing communication libraries' separation of user and communication buffer management leads to redundant data copies, increasing latency and HBM consumption. Achieving efficient user-buffer direct communication in supernode architectures is a pressing issue.
Innovation
StrataCL eliminates redundant data copies through registration-on-allocation. Its workload-balanced NPU-core partitioning and NPU-driven SDMA offloading fully leverage the high bandwidth and low latency of supernodes.
Methodology
- �� Registration-on-allocation: Asynchronous registration immediately after physical memory allocation.
- �� Shadow Virtual Addressing: Assign disjoint virtual address ranges to each NPU to simplify address translation.
- �� Workload-balanced partitioning: Avoids long-tail issues through NPU core partitioning.
- �� SDMA offloading: NPU cores submit DMA descriptors, and the SDMA engine asynchronously performs data movement.
Experiments
Experiments were conducted on Huawei CloudMatrix384 using up to 512 NPU dies and three real-world production applications. StrataCL's performance was compared with existing communication libraries in terms of bus bandwidth, inference throughput, and training time.
Results
StrataCL achieves up to 1.6x higher bus bandwidth for collective communication and 1.4x for MoE dispatch/combine. In LLM inference, throughput increases by 1.9x, and P99 TTFT is reduced by 2.2x. Training times are reduced by 1.4x and 1.3x, respectively.
Applications
StrataCL is applicable to distributed AI workloads requiring efficient communication, such as large-scale language model inference and training. Its efficient communication mechanisms can significantly enhance overall system performance.
Limitations & Outlook
StrataCL's scalability in large-scale scenarios needs improvement, especially under dynamic memory allocation patterns. Further validation of its applicability to different supernode architectures is needed.
Plain Language Accessible to non-experts
Imagine a large factory where machines need to quickly pass parts to each other. Traditional methods require parts to be placed in a staging area before being passed, wasting time and resources. StrataCL is like a direct conveyor belt between machines, allowing parts to be passed directly, greatly improving efficiency. This method not only reduces wait times but also conserves resources.
ELI14 Explained like you're 14
Imagine you're playing an online multiplayer game, and you need to quickly share gear with your teammates. The old way is like putting gear in a public box for teammates to pick up, wasting time. StrataCL is like having a direct conveyor belt between you and your teammates, making the game smoother and more fun!
Glossary
Supernode
A server architecture integrating numerous accelerators, offering high bandwidth and low latency communication.
StrataCL achieves efficient communication in supernode architectures.
Registration-on-allocation
Asynchronous registration immediately after memory allocation for direct user-buffer communication.
StrataCL eliminates redundant data copies through registration-on-allocation.
NPU-driven SDMA offloading
NPU cores submit DMA descriptors, and the SDMA engine asynchronously performs data movement.
Used to reduce NPU core contention and improve communication efficiency.
Shadow Virtual Addressing
Assigns disjoint virtual address ranges to each NPU to simplify address translation.
StrataCL simplifies communication operator implementation with shadow virtual addressing.
MoE Dispatch/Combine
An operation for large-scale model inference involving dynamic token routing.
StrataCL improves bus bandwidth in MoE dispatch/combine.
Open Questions Unanswered questions from this research
- 1 How to further optimize the scalability of registration-on-allocation under dynamic memory allocation patterns?
- 2 What is the applicability of StrataCL to other supernode architectures?
- 3 How to further reduce NPU core contention to improve compute-communication overlap?
Applications
Immediate Applications
Large-scale Language Model Inference
By improving communication efficiency, StrataCL can significantly enhance the performance of large-scale language model inference. Suitable for online services requiring quick responses.
Long-term Vision
Distributed AI System Optimization
StrataCL's communication optimization methods can be applied to other distributed AI systems, driving performance improvements across the industry.
Abstract
Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries remain largely buffer-centric because user and communication buffers are managed separately, causing redundant data copies or costly user-buffer registration. This paper presents StrataCL, a zero-redundancy and fabric-native communication library for production supernodes. StrataCL introduces registration-on-allocation to realize user-buffer direct communication, and designs communication operators with workload-balanced NPU-core partitioning and NPU-driven SDMA offloading to exploit supernode architecture features. On the Huawei CloudMatrix384, StrataCL improves collective bus bandwidth by up to 1.6x and improves MoE dispatch/combine bus bandwidth by up to 1.4x. Across three production workloads, StrataCL improves LLM inference throughput by 1.9x, reduces P99 TTFT by 2.2x, and reduces LLM and Recsys training iteration time by 1.4x and 1.3x, respectively.