DISC: A Dynamic Shape Compiler for Machine Learning Workloads
DISC compiler achieves dynamic shape optimization with up to 3.3x speedup.
Key Findings
Methodology
DISC utilizes the MLIR framework to design a dynamic shape compiler, extending HLO dialect with DHLO to support dynamic shapes. It generates runtime flow at compile time to avoid interpretation overhead and achieve host-device co-optimization. Kernel fusion is achieved through shape propagation and constraints collection.
Key Results
- DISC achieves up to 3.3x speedup over TensorFlow/PyTorch, with an average speedup of 2.27x across 6 popular models. Compared to Nimble, DISC achieves 1.8x speedup on the Transformer model.
- DISC shows an average speedup of 2.27x across 6 popular models, significantly outperforming existing methods.
- Efficient kernel fusion is achieved through shape constraints and propagation.
Significance
DISC provides an efficient compilation solution for dynamic shape workloads, significantly reducing compilation overhead and memory usage, simplifying optimization pipelines and deployment complexity. It offers new optimization insights for academia and industry, addressing long-standing challenges in dynamic shape model optimization.
Technical Contribution
DISC is the first to demonstrate how to build an end-to-end dynamic shape compiler based on MLIR infrastructure. It proposes a fully dynamic IR and compile-time generated runtime flow design, addressing kernel fusion problems without full shape information, and supports multiple machine learning frameworks.
Novelty
DISC is the first to efficiently support dynamic shapes with a fully dynamic IR and compile-time generated runtime flow design, offering significant innovation compared to existing methods.
Limitations
- DISC may perform poorly with dynamic rank, as dynamic rank is not commonly observed in practice.
- Optimization for specific shapes may not be as effective as static compilers.
- Further research is needed to implement optimizations on more devices.
Future Work
Future directions include exploring DISC's application on more devices and further optimizing dynamic shape processing efficiency. Authors suggest the community continue researching solutions to dynamic shape problems.
AI Executive Summary
Dynamic shape models are increasingly common in modern machine learning, yet existing compilers face challenges such as compilation overhead and memory usage when handling these models. The DISC compiler offers a solution through the MLIR infrastructure, supporting dynamic shape optimization. It extends the HLO dialect with DHLO, generating runtime flow to avoid interpretation overhead and achieve host-device co-optimization.
Experimental results show DISC achieves up to 3.3x speedup over TensorFlow/PyTorch, with an average speedup of 2.27x, significantly outperforming existing methods. It addresses kernel fusion problems through shape propagation and constraints collection, achieving more efficient optimization.
DISC's design provides new insights into optimizing dynamic shape models, with significant academic and industrial implications. Future directions include exploring its application on more devices and further optimizing dynamic shape processing efficiency.
Deep Dive
Abstract
Many recent machine learning models show dynamic shape characteristics. However, existing AI compiler optimization systems suffer a lot from problems brought by dynamic shape models, including compilation overhead, memory usage, optimization pipeline and deployment complexity. This paper provides a compiler system to natively support optimization for dynamic shape workloads, named DISC. DISC enriches a set of IR to form a fully dynamic shape representation. It generates the runtime flow at compile time to support processing dynamic shape based logic, which avoids the interpretation overhead at runtime and enlarges the opportunity of host-device co-optimization. It addresses the kernel fusion problem of dynamic shapes with shape propagation and constraints collecting methods. This is the first work to demonstrate how to build an end-to-end dynamic shape compiler based on MLIR infrastructure. Experiments show that DISC achieves up to 3.3x speedup than TensorFlow/PyTorch, and 1.8x than Nimble.