Towards a high-performance AI compiler with upstream MLIR
Achieved 90% of hand-written code performance using MLIR-based AI compiler.
Key Findings
Methodology
The study proposes an MLIR-based compilation flow using Linalg-on-Tensor IR from TensorFlow and PyTorch for cache-level optimization and micro-kernel lowering, achieving efficient vectorization. Key components include tensor dialect packing primitives, linear algebra pipeline, and micro-kernel lowering mechanism.
Key Results
- The compiler achieved over 90% performance of hand-written code, particularly excelling in multi-core environments.
- Utilizing VNNI, BFDOT, BFMMLA instructions, the compiler efficiently distributed tensors across various CPUs.
- In both single-thread and multi-thread settings, the compiler's performance matched hand-written code, especially on Intel and AMD architectures.
Significance
This research offers a novel compilation method for high-performance computing and machine learning, addressing existing compiler performance bottlenecks in complex linear algebra operations. It reduces reliance on manual optimization.
Technical Contribution
Contributions include new tensor packing and unpacking operations, improved upstream Linalg dialect compilation flow, and interface with libxsmm library, supporting optimizations across multiple CPU architectures.
Novelty
This is the first implementation of such an efficient linear algebra compilation flow in the MLIR framework, significantly enhancing compiler performance and flexibility compared to existing methods.
Limitations
- Performance may be limited by memory bandwidth in certain hardware configurations.
- Current optimization strategies may not fully exploit hardware features in some cases.
- Future work involves further optimizing the compiler's automated decision model.
Future Work
Future directions include optimizing the compiler's cost model, exploring support for more hardware architectures, and improving micro-kernel selection strategies.
AI Executive Summary
This study introduces a high-performance AI compiler based on MLIR, aiming to address the performance bottlenecks of existing compilers in handling complex linear algebra operations. Using Linalg-on-Tensor IR from TensorFlow and PyTorch, the compiler achieves cache-level optimization and micro-kernel lowering, excelling in multi-core environments.
The compiler's core technologies include tensor dialect packing primitives, linear algebra pipeline, and micro-kernel lowering mechanism. Experimental results show the compiler-generated code achieves over 90% performance of hand-written code, particularly excelling on Intel and AMD architectures.
This research offers a novel compilation method for high-performance computing and machine learning, reducing reliance on manual optimization and providing new directions for future compiler optimizations. Despite performance limitations in certain hardware configurations, further optimization of the compiler's cost model and automated decision strategies promise greater performance improvements in the future.
Deep Analysis
Background
With the advancement of machine learning and high-performance computing, compiler performance bottlenecks in handling complex linear algebra operations have become increasingly apparent. Existing compilers often rely on manual optimization and deep knowledge of specific hardware, limiting their generality and flexibility.
Core Problem
The core problem is achieving efficient linear algebra operation compilation without relying on manual optimization. This requires addressing cache optimization, micro-kernel selection, and multi-core performance bottlenecks.
Innovation
The study's core innovation lies in introducing an MLIR-based compilation flow that achieves automated cache optimization and micro-kernel lowering through tensor dialect packing primitives and a linear algebra pipeline.
Methodology
- �� Use Linalg-on-Tensor IR from TensorFlow and PyTorch
- �� Perform cache-level optimization to enhance data access efficiency
- �� Lower to micro-kernels for efficient vectorization
- �� Support optimizations across multiple CPU architectures
Experiments
The experimental design includes testing the compiler's performance on various CPU architectures using an MLP model for benchmarking, comparing compiler-generated code with hand-written code.
Results
Experimental results show the compiler achieves over 90% performance of hand-written code in multi-core environments, particularly excelling on Intel and AMD architectures.
Applications
The compiler can be used in high-performance computing and machine learning for linear algebra operations, reducing reliance on manual optimization and improving development efficiency.
Limitations & Outlook
Despite the compiler's excellent performance, it may be limited by memory bandwidth in certain hardware configurations. Future work involves further optimizing the compiler's automated decision model.
Plain Language Accessible to non-experts
Imagine a kitchen where a chef needs to quickly prepare multiple dishes. Traditional methods require the chef to manually chop and cook, while the new method uses automated equipment to quickly chop and cook. This compiler is like the automated equipment in the kitchen, optimizing processes to quickly handle complex computational tasks.
ELI14 Explained like you're 14
Imagine playing a game where you need to quickly upgrade your character. Traditional methods require you to manually choose equipment and skills, while the new method automatically selects the best equipment and skills for you. This compiler is like the game's auto-upgrade system, helping you quickly boost computational performance!
Glossary
MLIR (Multi-Level Intermediate Representation)
An intermediate representation framework for compiler development, supporting multi-level IR rewriting.
Used to implement the core optimization flow of the compiler.
Linalg-on-Tensor
An MLIR dialect for representing linear algebra operations.
Used as input IR for cache optimization and micro-kernel lowering.
VNNI (Vector Neural Network Instructions)
An instruction set for accelerating neural network computations.
Used to enhance the performance of compiler-generated code.
BFMMLA (BF16 Matrix Multiply Accumulate)
An instruction for BF16 data type matrix multiplication.
Implemented for efficient matrix operations on BF16-supporting CPUs.
libxsmm
An open-source micro-kernel library for high-performance computing.
The compiler interfaces with libxsmm through the XSMM dialect.
Open Questions Unanswered questions from this research
- 1 How to further optimize the compiler's automated decision model to enhance performance across different hardware configurations.
- 2 How to more effectively utilize hardware features on BF16-supporting CPUs.
Applications
Immediate Applications
High-Performance Computing
The compiler can be used to accelerate high-performance computing tasks, especially in handling complex linear algebra operations.
Long-term Vision
Machine Learning Optimization
By further optimizing the compiler's automated decision model, achieve more efficient machine learning model training and inference.
Abstract
This work proposes a compilation flow using open-source compiler passes to build a framework to achieve ninja performance from a generic linear algebra high-level abstraction. We demonstrate this flow with a proof-of-concept MLIR project that uses input IR in Linalg-on-Tensor from TensorFlow and PyTorch, performs cache-level optimizations and lowering to micro-kernels for efficient vectorization, achieving over 90% of the performance of ninja-written equivalent programs. The contributions of this work include: (1) Packing primitives on the tensor dialect and passes for cache-aware distribution of tensors (single and multi-core) and type-aware instructions (VNNI, BFDOT, BFMMLA), including propagation of shapes across the entire function; (2) A linear algebra pipeline, including tile, fuse and bufferization strategies to get model-level IR into hardware friendly tile calls; (3) A mechanism for micro-kernel lowering to an open source library that supports various CPUs.