Compiling ONNX Neural Network Models Using MLIR
Compiling ONNX models using MLIR improves inference efficiency.
Key Findings
Methodology
The paper presents the onnx-mlir compiler using MLIR infrastructure to generate inference code for ONNX models. Two new dialects are defined: ONNX-specific and loop-based dialects for graph-level and loop-level optimizations.
Key Results
- Experiments on IBM Power Systems with MNIST and ResNet50 models show significant inference performance improvements, especially in memory management and loop optimization.
- Compiled models demonstrate enhanced efficiency across architectures, notably on x86 and IBM System Z.
- Initial optimization work reveals substantial reductions in computational overhead through graph rewriting and constant propagation.
Significance
This research enhances ONNX model cross-platform performance through compiler technology, addressing deployment challenges in varied environments. It holds significant academic and industrial implications, especially for edge devices.
Technical Contribution
Introduces a compiler architecture based on MLIR, supporting multi-level intermediate representations, offering new theoretical guarantees and engineering possibilities. It significantly improves model portability and inference efficiency compared to existing methods.
Novelty
First to apply MLIR for ONNX model compilation, introducing two new dialects that optimize the inference process. Provides a more efficient code generation path compared to existing methods.
Limitations
- Currently supports a limited range of models, primarily popular deep learning models like MNIST and ResNet50.
- The compiler is still under development, with incomplete support for some architectures.
Future Work
Future work includes expanding supported model types, optimizing code generation across architectures, and further enhancing compiler stability and performance.
AI Executive Summary
Deep neural network models are increasingly popular in fields like computer vision, speech recognition, and natural language processing. To enhance model portability, the open-source community proposed the ONNX standard. This paper introduces the onnx-mlir compiler, which uses MLIR infrastructure to generate inference code for ONNX models. By defining two new dialects: ONNX-specific and loop-based dialects, it achieves graph-level and loop-level optimizations. Experimental results show significant inference performance improvements across architectures. Although still under development, the compiler supports popular models like MNIST and ResNet50. Future work will focus on expanding supported model types and optimizing code generation.
Deep Analysis
Background
Deep learning models have rapidly evolved over the past decade, particularly driven by GPU accelerators. The emergence of frameworks like Torch, Caffe, Theano, and TensorFlow accelerated this process. However, training and inference often occur in different environments, leading to model portability issues.
Core Problem
Deep learning models are typically trained in resource-rich environments, while inference may occur on edge devices. Existing methods face limitations in model rewriting and optimization, failing to meet diverse environmental demands.
Innovation
The paper proposes the onnx-mlir compiler, utilizing MLIR to generate inference code for ONNX models. It defines two new dialects: ONNX-specific and loop-based dialects, supporting graph-level and loop-level optimizations.
Methodology
- �� Utilize MLIR infrastructure for compiler architecture
- �� Define ONNX-specific dialect to encode standard semantics
- �� Create loop-based dialect for common lowering points
- �� Apply graph rewriting and constant propagation optimizations
Experiments
Experiments conducted on IBM Power Systems using MNIST and ResNet50 models. Evaluated compiler performance improvements across architectures, particularly in memory management and loop optimization.
Results
Results indicate significant efficiency improvements for compiled models on x86 and IBM System Z. Graph rewriting and constant propagation optimizations substantially reduce computational overhead.
Applications
The compiler can enhance inference performance of deep learning models on edge devices, particularly in resource-constrained environments.
Limitations & Outlook
Currently supports limited model types; the compiler is under development with incomplete support for some architectures. Future work will focus on expanding supported model types and optimizing code generation.
Plain Language Accessible to non-experts
Imagine you're in a kitchen preparing a big meal. You have a recipe (ONNX model) that needs to be cooked in different kitchens (devices). Our compiler acts like a versatile chef, adjusting ingredients and cooking steps (optimizations) based on each kitchen's characteristics, ensuring a delicious dish (efficient inference) every time.
ELI14 Explained like you're 14
Hey, friends! Imagine you're playing a super cool game. You have a character (ONNX model) that needs to complete tasks in different levels (devices). Our compiler is like a super helper, adjusting your character's gear and skills (optimizations) to ensure you win every level (efficient inference)!
Glossary
ONNX (Open Neural Network Exchange)
An open-source standard for exchanging neural network models, supporting interoperability across different frameworks.
Used to represent input model format in this paper.
MLIR (Multi-Level Intermediate Representation)
A modern compiler infrastructure supporting multi-level intermediate representations and optimizations.
Used to implement the compiler architecture.
Dialect
An intermediate representation defined in MLIR for domain-specific optimizations.
Two new dialects are defined in this paper: ONNX-specific and loop-based.
Optimization Pass
Processes in a compiler used to optimize code, such as graph rewriting and constant propagation.
Used to enhance model inference performance.
LLVM (Low-Level Virtual Machine)
An open-source project for compiler development, supporting code generation and optimization.
MLIR is a subproject of LLVM.
Open Questions Unanswered questions from this research
- 1 How to support more complex deep learning models.
- 2 How to further optimize code generation across different architectures.
Applications
Immediate Applications
Edge Device Inference Optimization
Enhance inference performance of deep learning models in resource-constrained environments, particularly on mobile and IoT devices.
Cross-Platform Model Deployment
Support efficient deployment of models across different hardware architectures, reducing development and maintenance costs.
Long-term Vision
Universal Compiler Architecture
Develop a universal compiler supporting various models and architectures, enhancing portability and performance of AI applications.
Abstract
Deep neural network models are becoming increasingly popular and have been used in various tasks such as computer vision, speech recognition, and natural language processing. Machine learning models are commonly trained in a resource-rich environment and then deployed in a distinct environment such as high availability machines or edge devices. To assist the portability of models, the open-source community has proposed the Open Neural Network Exchange (ONNX) standard. In this paper, we present a high-level, preliminary report on our onnx-mlir compiler, which generates code for the inference of deep neural network models described in the ONNX format. Onnx-mlir is an open-source compiler implemented using the Multi-Level Intermediate Representation (MLIR) infrastructure recently integrated in the LLVM project. Onnx-mlir relies on the MLIR concept of dialects to implement its functionality. We propose here two new dialects: (1) an ONNX specific dialect that encodes the ONNX standard semantics, and (2) a loop-based dialect to provide for a common lowering point for all ONNX dialect operations. Each intermediate representation facilitates its own characteristic set of graph-level and loop-based optimizations respectively. We illustrate our approach by following several models through the proposed representations and we include some early optimization work and performance results.