S2MAM: Semi-supervised Meta Additive Model for Robust Estimation and Variable Selection

Semi-supervised learning, a method that combines labeled and unlabeled data for learning, has gained widespread attention in data science. Manifold regularization is a classical semi-supervised learning framework that achieves learning by assuming the support of the unknown marginal distribution has the geometric structure of a Riemannian manifold. However, traditional manifold regularization methods rely on pre-specified similarity matrices, which can lead to inaccurate penalties when handling redundant or noisy input variables. To overcome these challenges, researchers have been exploring new methods to improve the robustness and adaptability of manifold regularization. In recent years, meta-learning and sparse additive models have gradually gained traction in machine learning, offering new ideas for solving complex data problems.

Traditional manifold regularization methods perform poorly when dealing with redundant and noisy variables, primarily because they rely on pre-specified similarity matrices. This approach can lead to inaccurate penalties when facing complex data, thus reducing predictive power. Moreover, existing methods often lack support for variable selection and model interpretability, limiting their applicability in real-world scenarios. Therefore, designing a new manifold regularization scheme that simultaneously achieves robustness, interpretability, and predictive effectiveness has become a pressing issue.

The core innovations of S2MAM lie in its bilevel optimization framework and probabilistic meta-learning strategy. First, S2MAM achieves automatic variable selection and similarity matrix updating through bilevel optimization, which contrasts sharply with traditional methods that pre-specify similarity matrices. Second, S2MAM employs a probabilistic meta-strategy to learn masks for input variables, enhancing model robustness and adaptability. Finally, S2MAM integrates manifold regularization with sparse additive models, enabling automatic variable masking and sparse approximation for high-dimensional inputs, even in the presence of noisy variables.

The methodology of S2MAM includes the following key steps:

• �� Bilevel Optimization Framework: The upper-level optimization is used to learn variable masks, while the lower-level optimization updates the decision function and similarity matrix.

• �� Probabilistic Meta-learning Strategy: A probabilistic meta-strategy is employed to learn masks for input variables, enhancing model robustness and adaptability.

• �� Manifold Regularization: Integrates manifold regularization with sparse additive models to achieve automatic variable masking and sparse approximation.

• �� Theoretical Guarantees: Provides theoretical guarantees for computational convergence and statistical generalization bounds, offering new possibilities for the design and optimization of semi-supervised learning models.

The experimental design includes validating S2MAM's effectiveness and robustness on four synthetic and twelve real-world datasets. The experiments used various baseline models, including LapSVM, f-FME, and AWSSL. Key hyperparameters were tuned via leave-one-out cross-validation to ensure optimal performance across different datasets. The experiments also included ablation studies to evaluate S2MAM's advantages in variable selection and model interpretability. The results demonstrate that S2MAM significantly improves prediction accuracy when handling redundant and noisy input variables.

The experimental results show that S2MAM excels in handling redundant and noisy input variables. For instance, on the Moon dataset, S2MAM maintains an accuracy above 89% even with noisy variables, whereas traditional LapSVM achieves only 55% under the same conditions. Additionally, on the ADNI clinical records dataset, S2MAM achieves an average MSE of approximately 0.119, significantly outperforming other baseline models. Ablation studies indicate that S2MAM has significant advantages in variable selection and model interpretability, automatically identifying truly informative variables and reducing the impact of noisy variables on model performance.

S2MAM is suitable for real-world applications that require handling large amounts of unlabeled data, such as medical imaging analysis and natural language processing. In these fields, data often contain a large number of redundant and noisy variables, which traditional manifold regularization methods struggle to handle effectively. By automating variable selection and updating the similarity matrix, S2MAM enhances model interpretability and predictive power, providing new tools for research and applications in these fields.

Despite S2MAM's outstanding performance in many aspects, it may face computational burdens when dealing with very large-scale datasets. Additionally, in certain specific noise conditions, S2MAM's performance may be affected, especially when noisy variables are highly correlated with informative variables. Future research directions include optimizing computational efficiency and enhancing the model's robustness under different noise conditions.