Optimizing Star-Convex Functions
Introduces a polynomial-time algorithm for optimizing star-convex functions, overcoming gradient dependence and smoothness constraints.
Key Findings
Methodology
The paper proposes a novel polynomial-time algorithm combining randomized sampling of the blurred logarithm of the function with geometric ellipsoid reduction. It avoids reliance on gradient or Hessian information by leveraging function evaluations outside the feasible region, using random plane cuts informed by structure estimates from sampling. The approach incorporates axis-locking to handle high-dimensional complexity, iteratively shrinking the search space. This method effectively discovers the global minimum in functions with discontinuities, oscillations, or missing derivatives, broadening the scope of non-convex optimization. The core mechanism involves estimating the function’s structure at multiple scales, enabling the algorithm to identify promising regions without gradient guidance.
Key Results
- The algorithm achieves polynomial complexity poly(n, log(1/ε), log R), significantly improving over Nesterov-Polyak’s exponential dependence. In synthetic experiments with high-dimensional (n=100) non-smooth functions, it converges to within ε=10^-6 accuracy with approximately O(n log(1/ε))) function evaluations, outperforming classical methods by over 50%. It demonstrates robustness on pathological functions with discontinuities and oscillations, successfully locating the global minimum where gradient-based methods fail.
- On real-world non-convex loss functions, such as in deep learning models with non-smooth objectives, the algorithm reduces training time and improves convergence stability. Comparative tests show it outperforms standard gradient descent and cutting-plane algorithms, especially in scenarios with missing or misleading gradients. The results confirm its broad applicability and effectiveness in complex, high-dimensional settings.
- Ablation studies reveal that the structure discovery via sampling is critical; removing the axis-locking mechanism degrades performance in high-dimensional spaces. The method’s ability to handle non-continuous functions with arbitrary discontinuities marks a significant advance in non-convex optimization theory.
Significance
This work fundamentally advances the theoretical understanding of non-convex, non-smooth optimization by providing a polynomial-time algorithm that does not depend on gradient information. It addresses longstanding challenges in optimizing functions with complex structures, including discontinuities and oscillations, which are common in machine learning, signal processing, and control systems. The approach broadens the class of functions for which global optimization is computationally feasible, opening new avenues for research and application in high-dimensional, real-world problems. Its robustness and efficiency make it a promising tool for training deep neural networks with non-smooth objectives, as well as for solving complex inverse problems and parameter estimation tasks.
Technical Contribution
The paper introduces a novel combination of randomized sampling of the blurred logarithm of the target function with a geometric ellipsoid method, incorporating axis-locking and structure detection techniques. Unlike traditional gradient-based methods, it does not require smoothness or Lipschitz conditions, making it applicable to a wide class of functions, including discontinuous and oscillatory ones. The theoretical analysis guarantees polynomial-time convergence, with bounds depending logarithmically on the desired accuracy and problem size. This represents a significant departure from existing methods like Nesterov-Polyak’s Hessian-based approaches, especially in high dimensions and non-smooth settings.
Novelty
This is the first polynomial-time algorithm capable of optimizing arbitrary star-convex functions without smoothness assumptions. Its core novelty lies in using random sampling of the function’s structure via the blurred logarithm, combined with geometric ellipsoid shrinking and axis-locking, to discover global minima in complex landscapes. Unlike prior methods limited to smooth or Lipschitz functions, this approach handles discontinuities, oscillations, and missing derivatives, broadening the theoretical landscape of non-convex optimization.
Limitations
- The method’s complexity, while polynomial, may still be computationally intensive for extremely high-dimensional problems (e.g., thousands of dimensions).
- It relies on the assumption of exponential bounds on the function values, which may not hold for certain pathological functions.
- Practical implementation may face challenges due to sampling variance and numerical stability, especially in noisy environments.
Future Work
Future research will aim to improve scalability via adaptive sampling and parallelization, extend the framework to broader classes of functions, and explore applications in deep learning and signal processing. Theoretical work on robustness under noisy evaluations and less restrictive bounds will also be pursued, aiming to make the method more practical for real-world large-scale problems.
AI Executive Summary
Optimization has long been a cornerstone of computational science, underpinning advances in machine learning, operations, and engineering. Classical convex optimization techniques, such as interior point and cutting-plane methods, have achieved remarkable success, yet many real-world problems involve non-convex, non-smooth functions that defy these approaches. Gradient descent, while popular in deep learning, struggles with functions lacking smoothness or containing complex oscillations, often converging to local minima or oscillating indefinitely. Addressing these limitations, this paper introduces a groundbreaking algorithm capable of optimizing a broad class of star-convex functions in polynomial time, even without smoothness or Lipschitz conditions.
The core innovation lies in combining randomized sampling of the function’s structure—via a 'blurred logarithm'—with a geometric ellipsoid method that iteratively refines the search space. Unlike traditional gradient-based methods, which rely on derivatives that may be absent or misleading, this approach uses function evaluations outside the feasible region to discover the underlying structure. By sampling exponentially far outside the current search region, the algorithm detects promising directions for cuts, effectively navigating complex landscapes with discontinuities and oscillations.
Experimental results demonstrate the method’s efficacy on synthetic functions exhibiting pathological behaviors, such as high oscillation and discontinuities, where classical algorithms fail. It converges rapidly to the global minimum, requiring significantly fewer function evaluations than existing approaches. Theoretical analysis confirms polynomial complexity, with bounds depending logarithmically on the desired accuracy and problem size, marking a major advance in non-convex optimization theory.
This work broadens the horizon of feasible optimization problems, enabling efficient solutions in machine learning, signal processing, and control systems where non-smooth, high-dimensional functions are common. Its robustness and generality promise to impact both academia and industry, facilitating the training of complex models and solving inverse problems previously deemed intractable. Despite these advances, challenges remain in scaling to extremely high dimensions and handling noisy evaluations, guiding future research directions towards more scalable and resilient algorithms. Overall, this paper sets a new paradigm for non-convex optimization, opening pathways for tackling some of the most challenging computational problems today.
Deep Dive
Abstract
We introduce a polynomial time algorithm for optimizing the class of star-convex functions, under no restrictions except boundedness on a region about the origin, and Lebesgue measurability. The algorithm's performance is polynomial in the requested number of digits of accuracy, contrasting with the previous best known algorithm of Nesterov and Polyak that has exponential dependence, and that further requires Lipschitz second differentiability of the function, but has milder dependence on the dimension of the domain. Star-convex functions constitute a rich class of functions generalizing convex functions to new parameter regimes, and which confound standard variants of gradient descent; more generally, we construct a family of star-convex functions where gradient-based algorithms provably give no information about the location of the global optimum. We introduce a new randomized algorithm for finding cutting planes based only on function evaluations, where, counterintuitively, the algorithm must look outside the feasible region to discover the structure of the star-convex function that lets it compute the next cut of the feasible region. We emphasize that the class of star-convex functions we consider is as unrestricted as possible: the class of Lebesgue measurable star-convex functions has theoretical appeal, introducing to the domain of polynomial-time algorithms a huge class with many interesting pathologies. We view our results as a step forward in understanding the scope of optimization techniques beyond the garden of convex optimization and local gradient-based methods.