Diversed Model Discovery via Structured Table Discovery

The proliferation of machine learning models has led to the emergence of model lakes—centralized repositories that store heterogeneous models alongside their metadata and documentation. Model cards have become a standard artifact within these lakes, encapsulating critical information such as training datasets, evaluation results, configurations, and intended usage. These cards facilitate transparency and reproducibility, enabling users to understand and compare models effectively. Existing model search systems, including platforms like HuggingFace and ModelScope, primarily treat model cards as unstructured text documents, employing keyword search, metadata filtering, or semantic retrieval techniques. While effective for locating individually relevant models, these approaches often yield homogeneous results clustered around dominant model families due to shared writing templates and stylistic conventions, limiting exploration of alternative models. Concurrently, data lake research has developed sophisticated table discovery and integration techniques, such as keyword search, joinability, and unionability, to locate and consolidate relevant tabular data across heterogeneous sources. These advances inspire the integration of structured table discovery into model card retrieval, aiming to leverage the high-density, decision-critical evidence encapsulated in performance and configuration tables. Additionally, information retrieval research has introduced nugget-based evaluation methods that decompose answers into atomic evidence units, addressing the limitations of document-level relevance metrics and enabling fine-grained assessment of coverage and diversity. These interdisciplinary insights set the stage for a novel approach to model search that balances task alignment with diversity through structured evidence.

The core problem addressed is the challenge of retrieving a set of machine learning models from a large, heterogeneous model lake that are not only relevant to a user’s task but also diverse in measurable ways such as architecture, training corpus, evaluation benchmarks, and performance trade-offs. Traditional semantic similarity-based retrieval methods optimize for textual proximity, often resulting in retrieval sets dominated by closely related models with similar narrative descriptions, thereby limiting the user’s ability to explore alternative approaches. This homogeneity is exacerbated by shared authorial styles and reporting conventions. Moreover, textual descriptions in model cards tend to be verbose, heterogeneous, and influenced by author style, complicating direct comparison. In contrast, structured tables within model cards provide condensed, high-quality evidence that varies meaningfully even among related models. However, existing search systems do not leverage this structured evidence effectively. Additionally, model lakes are dynamic, continuously growing repositories, making fixed gold-standard annotations for evaluation impractical. Therefore, there is a need for a retrieval framework that balances task alignment and diversity by utilizing structured table evidence, coupled with an evaluation protocol that is query-aware, evidence-oriented, and scalable to dynamic model lakes.

This work introduces several key innovations:

• �� Integration of semantic retrieval with structured table discovery operators—keyword search, joinability, and unionability—from the Blend system, enabling expansion of candidate model sets beyond text-based similarity and addressing result homogeneity.

• �� Development of an orientation-aware table integration algorithm that automatically detects and corrects table transpositions and partial overlaps, producing compact, coherent integrated views that enhance evidence comparability and user interpretability.

• �� Design of a nugget-based, dual-stage evaluation protocol that extracts fixed-schema structured evidence tuples (model, base model, variant, dataset, metric name, metric value) from model cards using prompt-based extraction, and maps queries to nugget constraints via prompt-assisted filtering. This enables fine-grained, auditable measurement of evidence coverage and diversity across retrieved candidate sets, supporting dynamic and scalable evaluation in evolving model lakes.

These innovations collectively advance model search from unstructured text retrieval to a structured, evidence-based paradigm that better aligns with user needs for comparative and diverse model discovery.

• �� Semantic Retrieval (NL2Card): Employ Sentence-BERT encoder with FAISS for dense retrieval, complemented by sparse retrieval via Pyserini and a hybrid approach combining both, serving as baseline methods.

• �� Table Discovery Pipeline (NL2Card2Tab2Card): Starting from an anchor model card retrieved semantically, extract associated tables as anchor tables.

• �� Keyword Table Search: Construct keyword queries from anchor table headers and first columns; use Blend’s value-based keyword search operator to retrieve candidate tables ranked by token match frequency.

• �� Joinable Table Search: Use the anchor table’s first column as a query column to find tables joinable on shared entities (e.g., model names, datasets), ranked by overlap size.

• �� Unionable Table Search: Identify tables with compatible schemas for union operations, ranked by the number of alignable columns.

• �� Mapping Tables to Model Cards: For each retrieved table, select the single model card with the highest semantic similarity to the query, ensuring one representative card per table to avoid redundancy.

• �� Orientation-aware Table Integration: Detect whether tables are transposed by comparing header keywords with first columns; transpose as needed before integrating tables to handle partial overlaps and heterogeneity, producing compact integrated views.

• �� Nugget Extraction: Use prompt-based models to extract fixed six-attribute tuples (model, base model, variant, dataset, metric name, metric value) from heterogeneous model card content, normalizing evidence.

• �� Query-to-Nugget Mapping: Map queries to nugget attribute constraints using prompt-based filtering, accommodating vague and specific queries.

• �� Candidate Set Scoring: Compute the count of unique query-relevant nuggets covered by the retrieved candidate set, measuring evidence coverage and diversity independent of ranking order, suitable for dynamic model lakes.

Experiments utilize a model lake comprising over 60,000 HuggingFace model cards and their associated structured tables, filtered to retain compact tables with fewer than 200 rows and 100 columns to ensure information density. The query set consists of 597 model recommendation queries adapted from a scientific literature retrieval benchmark via prompt-based rewriting to preserve original intent while shifting focus from papers to models. Baselines include dense, sparse, and hybrid semantic retrieval methods. Evaluation employs the nugget-based coverage metric, quantifying the number of unique query-relevant evidence tuples retrieved. Additional qualitative evaluation assesses the effectiveness of orientation-aware table integration in producing coherent, comparable views. Ablation studies analyze the contribution of each table discovery operator (keyword, joinable, unionable) to overall performance. The evaluation framework supports dynamic model lake growth by requiring nugget extraction only upon ingestion of new model cards, ensuring scalability and stability.

The structure-aware pipeline consistently outperforms semantic baselines across 597 queries in terms of nugget coverage, retrieving a broader and more diverse set of evidence encompassing model variants, datasets, and performance metrics. Orientation-aware table integration successfully resolves transposition and partial overlap issues, yielding compact integrated views that facilitate user comparison. The combined use of keyword, joinable, and unionable table discovery operators significantly expands the candidate model set, mitigating the homogeneity inherent in text-only retrieval. Ablation experiments confirm that omitting any operator reduces coverage, underscoring their complementary roles. The approach demonstrates robustness and scalability in a dynamic, large-scale model lake environment, supporting continuous updates and automated evaluation.

This framework is directly applicable to large-scale model lakes for efficient and diverse model discovery, aiding researchers and practitioners in identifying task-aligned models with varied characteristics to inform selection and deployment. It can be integrated into model management platforms to enhance model card quality through structured evidence extraction and consolidation, improving documentation standardization and usability. The approach also facilitates scalable, evidence-based evaluation and benchmarking of models in dynamic repositories. Future extensions may incorporate multimodal data sources such as code and model weights, advancing intelligent model ecosystem management and application.

The method’s effectiveness depends on the availability and quality of structured tables within model cards; incomplete or inconsistent tables limit retrieval performance. The Blend-based table discovery operators may face computational and scalability challenges when applied to extremely large model lakes. Nugget extraction and query mapping rely on prompt-based models, which may have reduced accuracy with diverse formatting and semantically ambiguous queries. The current framework does not incorporate user feedback or personalization, which could enhance adaptability and relevance. Additionally, handling conflicting or inconsistent evidence across model cards remains an open challenge.