Data Valuation and Selection in a Federated Model Marketplace

The evolution of AI has driven a surge in demand for large-scale, high-quality datasets. Early AI development relied heavily on web data, leaving many valuable industry datasets underutilized. Recently, data marketplaces have emerged as crucial platforms to facilitate cross-sector data sharing, especially in regulated fields like finance and healthcare. Traditional data transactions are centralized, raising privacy, traceability, and ethical concerns. Federated learning (FL) offers a decentralized approach, enabling collaborative model training without raw data exchange. Despite its promise, FL faces challenges such as data heterogeneity, performance prediction, and privacy preservation. Existing metrics often fail to accurately estimate model performance across diverse, non-i.i.d. data sources, limiting effective data valuation and selection. Addressing these issues is vital for building trustworthy, efficient, and scalable data markets.

The core challenge lies in accurately predicting model performance across heterogeneous data sources without direct data access. Existing divergence metrics are insufficient in non-i.i.d. FL settings, where data distributions vary significantly. Moreover, selecting the optimal data mixture to maximize model accuracy under privacy constraints is computationally expensive, especially with many data sources. How to reliably evaluate data quality, guide data acquisition, and ensure model robustness in such environments remains an open problem. These issues hinder the development of trustworthy, scalable federated data markets and limit the practical deployment of FL systems in real-world scenarios.

This work introduces several innovations: 1) CombineWad, a Wasserstein-distance-based predictor that estimates model performance by measuring distribution discrepancies among multiple data sources; 2) a distributed, privacy-preserving algorithm for Wasserstein distance approximation, avoiding raw data sharing; 3) integration of neural scaling laws to project small-scale trial results to larger data scales, reducing computational costs; 4) a performance-guided data mixture optimization method that dynamically adjusts source contributions for maximal accuracy. These innovations collectively address the limitations of existing divergence metrics, enabling accurate, privacy-preserving performance prediction and data selection in complex FL environments.

• �� Use Wasserstein distance to quantify distribution differences between data sources and validation sets, forming the basis for performance prediction. • Conduct small-scale federated trial runs on pilot data, measuring distances and model accuracy, to learn the relationship between distribution discrepancy and performance. • Develop a distributed algorithm leveraging geometric properties to approximate Wasserstein distance without raw data sharing, ensuring privacy. • Incorporate neural scaling laws to project performance from small pilot experiments to large-scale data scenarios, enabling cost-effective prediction. • Use gradient-based optimization to iteratively adjust data source proportions, maximizing predicted performance. • Implement privacy-preserving distance estimation by sharing interpolated measures and cost matrices, avoiding raw data exchange. • Validate the framework through extensive experiments on datasets like CIFAR-10, MNIST, and medical images, comparing with baseline divergence metrics, and analyzing robustness under label skew, mislabeling, and unlabeled data.

The experiments involve multiple datasets—CIFAR-10, MNIST, Fashion-MNIST, ImageNet, and RSNA Bone Age—covering diverse data distributions and tasks. Baseline FL algorithms (FedAvg, FedProx, Scaffold, FedNova) are evaluated under label skew, mislabeling, and unlabeled scenarios. The key metrics include accuracy, convergence speed, and distance estimation error. Pilot experiments with limited data validate the distance approximation accuracy and performance projection. The framework's ability to identify optimal data mixtures and predict performance across scales is tested through ablation studies and cross-scenario comparisons, demonstrating robustness and efficiency.

CombineWad achieves over 85% accuracy in performance prediction across datasets, outperforming traditional divergence metrics by 20%. It reduces training time by 50% while maintaining high accuracy. The privacy-preserving distance approximation maintains errors below 5%, suitable for practical deployment. The method effectively guides data mixture selection, improving convergence speed by 20% and model accuracy by over 10% in heterogeneous environments. These results confirm the framework’s effectiveness in real-world federated data markets.

This framework can be directly applied in financial, healthcare, autonomous driving, and other industries where data privacy and heterogeneity are critical. It enables data providers and buyers to evaluate data quality, optimize data acquisition, and trade models efficiently. The approach supports building trustworthy, scalable federated systems, facilitating data-driven decision-making while safeguarding privacy. Long-term, it can underpin real-time performance monitoring, adaptive data scheduling, and multi-task federated learning, transforming AI deployment in complex, privacy-sensitive environments.

Distance approximation accuracy may decline in highly heterogeneous or high-dimensional data, affecting performance estimates. Scalability in extremely large data sources or under strict privacy constraints remains challenging. The reliance on neural scaling laws assumes their universal applicability, which may not hold for all tasks, potentially biasing predictions. Future work should focus on improving approximation robustness, extending to dynamic data streams, and optimizing algorithms for large-scale, real-time applications.