Structural interpretability in SVMs with truncated orthogonal polynomial kernels

Support Vector Machines (SVMs) have long been a cornerstone in binary classification due to their robust theoretical foundation and the flexibility provided by kernel methods. By replacing inner products with suitable kernels, SVMs can create nonlinear decision boundaries while maintaining the convexity of the optimization problem. However, once a kernel is fixed and the model is trained, the resulting classifier is often difficult to interpret beyond standard predictive metrics, especially in nonlinear settings. Traditionally, interpreting the complexity of SVMs often relies on surrogate models or local explanation schemes, which typically require additional optimization steps and may fail to capture the global structure of the model. In recent years, as interpretability has become increasingly important in machine learning, researchers have begun to explore new methods to reveal the internal structure of complex models.

In nonlinear settings, the decision function of SVMs is represented implicitly through kernel evaluations rather than through a small collection of directly readable coefficients, making interpretation challenging. Particularly when using complex kernel functions, traditional interpretation methods may fail to reveal the internal structure of the model. How to extract structural information from a trained SVM without relying on surrogate models or retraining has become a significant research problem. This not only involves understanding model complexity but also relates to how to optimize model performance, especially in applications where detailed analysis of model decisions is required.

The core innovation of this paper lies in using truncated orthogonal polynomial kernels to analyze the structural complexity of SVMs. By employing an explicit tensor-product orthonormal basis in a finite-dimensional reproducing kernel Hilbert space (RKHS), researchers can precisely expand the trained decision function. This method introduces the Orthogonal Representation Contribution Analysis (ORCA) framework, which uses Orthogonal Kernel Contribution (OKC) indices to quantify how the squared RKHS norm of the classifier is distributed across interaction orders, total polynomial degrees, marginal coordinate effects, and pairwise contributions. The innovation of this method lies in its ability to provide precise quantification of model complexity, revealing intrinsic geometric features of the model structure.

• �� Construct SVM models using truncated orthogonal polynomial kernels, ensuring the reproducing kernel Hilbert space (RKHS) is finite-dimensional.
• �� Use an explicit tensor-product orthonormal basis in the RKHS to expand the trained decision function.
• �� Introduce the Orthogonal Representation Contribution Analysis (ORCA) framework, quantifying the distribution of the classifier's squared RKHS norm through Orthogonal Kernel Contribution (OKC) indices.
• �� Validate the method's effectiveness through experiments on a synthetic double-spiral problem and a real five-dimensional echocardiogram dataset.
• �� Analyze the distribution of OKC indices across different interaction orders and polynomial degrees, revealing model complexity.

The experimental design involves validating the proposed method on a synthetic double-spiral problem and a real five-dimensional echocardiogram dataset. Synthetic data is used to test the method's performance on known structures, while real data evaluates its effectiveness in practical applications. In the experiments, researchers used different truncated orthogonal polynomial kernels and analyzed the distribution of Orthogonal Kernel Contribution (OKC) indices across different interaction orders and polynomial degrees. Through these experiments, researchers were able to verify the effectiveness of the proposed method and reveal structural aspects of model complexity.

The experimental results demonstrate that the proposed Orthogonal Representation Contribution Analysis (ORCA) framework effectively reveals structural aspects of model complexity not captured by predictive accuracy alone. In the synthetic double-spiral problem, OKC indices show whether the model is primarily driven by marginal effects or interactions. In the real five-dimensional echocardiogram dataset, researchers observed whether most of the model's norm is concentrated in low-degree modes or spread over higher degrees. These results indicate that the proposed method provides deep insights into the model's internal structure.

The method's application scenarios include fields requiring high interpretability, such as medical diagnosis, financial risk analysis, and autonomous driving. In these fields, understanding the decision-making process of models is crucial for ensuring their reliability and safety. By providing detailed quantification of model complexity, the method can help researchers and practitioners optimize model performance and make adjustments when necessary. Additionally, the method can be used for model debugging and optimization, helping to identify and resolve potential issues.

Although the method provides detailed analysis of model complexity, its effectiveness in practical applications needs further validation, especially across different domains and datasets. Additionally, the method relies on the choice of truncated orthogonal polynomial kernels, which may not be suitable for all types of datasets or problem domains. For some complex high-dimensional datasets, the choice of orthogonal basis may affect the interpretability of the model. In high-dimensional spaces, although the orthogonal basis provides precise expansion, the computational complexity may significantly increase, limiting the method's applicability to large-scale datasets.