On subspace-constrained preconditioning for randomized iterative methods

The evolution of solving large linear systems has seen classical methods like Krylov subspace algorithms, including CG and GMRES, which excel under well-conditioned matrices. However, their performance deteriorates with increasing condition numbers. Randomized methods such as the randomized Kaczmarz and stochastic gradient descent have gained popularity for their low memory footprint and scalability. Recent advances introduced subspace constraints to improve convergence, but these often relied on full-rank assumptions, limiting their scope. Addressing ill-conditioned and rank-deficient systems remains a challenge, motivating the development of more flexible, theoretically grounded approaches.

The main challenge is to enhance the convergence rate of randomized iterative methods without restrictive assumptions like full rank. Existing strategies struggle with ill-conditioned or sparse matrices, where spectral properties hinder rapid convergence. Constructing effective preconditioners is computationally expensive, especially for large systems. Therefore, designing a universal, efficient preconditioning framework that guarantees linear convergence, handles rank deficiencies, and avoids explicit preconditioner formation is critical for advancing large-scale linear system solvers.

The paper introduces a QR-like factorization that decomposes the transpose of the coefficient matrix into block-orthogonal components, enabling subspace constraints without full-rank assumptions. This reduces the problem to a smaller, better-conditioned system. It combines stochastic gradients with orthogonal search directions, accelerating convergence. The method is implemented implicitly, avoiding costly preconditioner construction, and provides explicit convergence guarantees. These innovations collectively extend the applicability of randomized methods, especially for ill-conditioned and rank-deficient matrices, with theoretical and empirical validation.

• �� Construct a QR-like factorization of \(A^T\) based on a fixed submatrix \(A_{Ip}\), yielding an orthogonal block structure. • Enforce the subsystem \(A_{Ip} x = b_{Ip}\) throughout iterations, reducing the original problem to a smaller one with favorable spectral properties. • Use stochastic gradients to generate search directions, then apply Gram-Schmidt orthogonalization to produce orthogonal search directions for acceleration. • Develop an implicit implementation that avoids explicit preconditioner formation, leveraging the factorization and subspace constraints. • Theoretically prove linear convergence in expectation, deriving bounds based on spectral properties of the reduced system. • Validate through experiments on synthetic and real datasets, comparing convergence speed and robustness.

Experiments involve synthetic matrices with varying condition numbers and real datasets from signal processing. Baselines include classical Kaczmarz, unpreconditioned stochastic gradient, and existing subspace methods. Metrics focus on iteration count, convergence rate, and computational time. Hyperparameters such as subspace size and sampling distribution are tuned. Ablation studies assess the impact of orthogonal search directions and QR-based factorization. Results show a 30% average acceleration in convergence, with improved stability in rank-deficient and ill-conditioned cases, confirming theoretical predictions.

The proposed method achieves over 30% faster convergence compared to traditional randomized methods, especially in systems with condition numbers exceeding 10^8. Incorporating orthogonal search directions further reduces iteration counts by 20%. The implicit implementation maintains low computational overhead, suitable for high-dimensional problems. The spectral analysis confirms that the reduced system has a more favorable singular value distribution, directly translating into faster convergence. These results demonstrate the method's robustness, efficiency, and broad applicability.

Applicable in large-scale machine learning, signal reconstruction, and scientific simulations where high-dimensional linear systems are prevalent. Its ability to handle rank deficiencies and ill-conditioned matrices makes it valuable for real-world data analysis, especially in scenarios demanding fast, reliable solutions without heavy preconditioning costs. The implicit scheme facilitates integration into distributed computing environments, enabling scalable high-performance computing.

The effectiveness depends on the choice of subspace \(Ip\); poor selection may limit acceleration. Handling extremely sparse or highly ill-conditioned matrices still poses challenges. Variance introduced by stochastic sampling requires careful tuning. Future work should focus on adaptive subspace selection, variance reduction, and extending the framework to nonlinear and non-convex problems.