Neo: A Learned Query Optimizer
Neo employs deep neural networks for end-to-end query optimization, achieving performance comparable to commercial systems.
Key Findings
Methodology
Neo integrates imitation learning from traditional optimizers, utilizing tree convolutional neural networks to model query plans, combined with reinforcement learning and demonstration learning for continuous improvement. The system encodes queries via adjacency matrices and attribute vectors, and plans via tree-structured embeddings. It employs a value network to predict plan latency, guiding a best-first search for optimal plans. The training involves collecting execution data, updating the neural models iteratively, and adapting to data patterns and estimation errors, enabling robust, holistic optimization.
Key Results
- Starting from PostgreSQL, Neo achieves performance on par with Oracle and Microsoft SQL Server, with query latency improvements of 15%-20% across multiple complex workloads. Training time is reduced from days to hours, demonstrating high learning efficiency and generalization. Neo adapts to inaccurate cardinality estimates and user preferences, such as worst-case or average-case optimization, showing flexibility and robustness.
- In unseen query scenarios, Neo generalizes well, maintaining strong performance and robustness. It effectively handles multi-table joins, index selection, and operator choices, outperforming traditional heuristics and existing learned methods in key metrics. The system's ability to learn from real execution feedback accelerates convergence and improves plan quality.
- Experimental results confirm Neo’s capability to surpass baseline optimizers, with consistent performance gains in diverse workloads, validating the feasibility of fully learned query optimization at scale.
Significance
This work marks a milestone in database research by demonstrating that a fully end-to-end learned query optimizer can match and sometimes exceed the performance of established commercial systems. It addresses longstanding challenges of manual tuning, heuristic limitations, and maintenance complexity, paving the way for fully automated, adaptive database management. The approach reduces development costs, enhances system flexibility, and opens new avenues for integrating AI-driven decision-making into data systems, ultimately transforming how databases are designed, maintained, and optimized in the era of big data.
Technical Contribution
The paper introduces a comprehensive framework combining query representation, plan encoding, a deep value network, and reinforcement learning-guided search. It innovatively applies tree convolutional neural networks to process plan structures, employs multi-layer feature engineering for semantic richness, and leverages demonstration learning for rapid initialization. The system replaces handcrafted cost models with neural predictors, enabling holistic optimization. This integration of deep learning and reinforcement learning into the core of query planning constitutes a significant advancement over prior partial or heuristic-based methods, offering a scalable, adaptable solution.
Novelty
This is the first system to realize a fully end-to-end learned query optimizer capable of generating complete execution plans. Unlike previous works focusing on sub-tasks like cardinality estimation or join ordering, Neo unifies these components into a single learning framework. Its use of tree-structured neural networks, combined with reinforcement and demonstration learning, represents a novel paradigm shift in database optimization, setting a new benchmark for research and practical deployment.
Limitations
- Neo currently depends on traditional optimizers for initial demonstration data, limiting full autonomy. Its support for complex query types, such as nested subqueries, remains limited. The model's generalization across different database schemas and data distributions needs further validation.
- Training requires substantial demonstration data and computational resources, which may hinder scalability. The system's robustness under extreme estimation errors or data shifts is not yet fully established, necessitating further research into more adaptive features and models.
- While promising, the approach still faces challenges in handling dynamic data updates and real-time query workloads, which require ongoing adaptation and efficiency improvements.
Future Work
Future directions include developing fully autonomous learning mechanisms that do not rely on initial demonstrations, enhancing transferability across diverse database schemas, and supporting more complex query types. Incorporating hardware-aware optimization and real-time adaptation will be crucial for practical deployment. Further, integrating explainability features and robustness against data shifts will improve system reliability. The community is encouraged to explore multi-task learning, meta-learning, and scalable training strategies to accelerate progress toward fully autonomous, adaptive database systems.
AI Executive Summary
Query optimization has long been a cornerstone of database performance, traditionally relying on handcrafted heuristics and cost models designed by experts. While effective, these methods are labor-intensive, brittle, and difficult to adapt to evolving workloads and data distributions. Recent advances in machine learning have shown promise in improving specific sub-tasks such as cardinality estimation or join ordering, but a comprehensive, end-to-end solution has remained elusive.
This paper introduces Neo, a pioneering system that leverages deep neural networks, reinforcement learning, and demonstration learning to create a fully learned query optimizer. Neo begins by collecting experience from a traditional optimizer, such as PostgreSQL, generating a dataset of query plans and their execution latencies. It then trains a deep neural network, specifically employing tree convolutional architectures, to predict the latency of partial and complete query plans. This value model guides a best-first search through the plan space, enabling Neo to generate high-quality execution plans.
The core innovation lies in integrating multiple components—query representation, plan encoding, value prediction, and search—into a cohesive, end-to-end learning framework. By continuously updating its models based on real execution feedback, Neo adapts to data patterns and estimation errors, demonstrating robust performance. Experimental results show that Neo, even when bootstrapped from a simple optimizer like PostgreSQL, can achieve performance comparable to or better than leading commercial systems like Oracle and Microsoft SQL Server. In several benchmarks, Neo reduces query latency by 15%-20%, with training times shortened from days to hours.
This breakthrough has profound implications for the future of database management. It suggests that fully autonomous, data-driven optimization is feasible, reducing reliance on manual tuning and heuristic rules. Neo’s ability to adapt and improve over time paves the way for more flexible, maintainable, and scalable data systems. Despite current limitations—such as dependence on demonstration data and support for specific query types—the framework offers a promising blueprint for next-generation intelligent databases. Future work will focus on enhancing autonomy, generalization, and real-time adaptation, ultimately transforming how data systems are designed and operated in the era of big data.
Deep Dive
Abstract
Query optimization is one of the most challenging problems in database systems. Despite the progress made over the past decades, query optimizers remain extremely complex components that require a great deal of hand-tuning for specific workloads and datasets. Motivated by this shortcoming and inspired by recent advances in applying machine learning to data management challenges, we introduce Neo (Neural Optimizer), a novel learning-based query optimizer that relies on deep neural networks to generate query executions plans. Neo bootstraps its query optimization model from existing optimizers and continues to learn from incoming queries, building upon its successes and learning from its failures. Furthermore, Neo naturally adapts to underlying data patterns and is robust to estimation errors. Experimental results demonstrate that Neo, even when bootstrapped from a simple optimizer like PostgreSQL, can learn a model that offers similar performance to state-of-the-art commercial optimizers, and in some cases even surpass them.