Diagnosable ColBERT: Debugging Late-Interaction Retrieval Models Using a Learned Latent Space as Reference

In the field of biomedical and clinical information retrieval, reliable retrieval requires more than strong ranking performance; it requires a practical method to identify systematic model failures and curate training evidence to correct them. Existing late-interaction models like ColBERT provide an initial solution by exposing interpretable interaction scores between document and query tokens. However, this interpretability is shallow: it explains a specific document-query pair score but does not reveal whether the model has learned clinical concepts in a stable, reusable, and context-sensitive manner across diverse expressions. As a result, these scores offer limited support for diagnosing misunderstandings, identifying unreasonably distant biomedical concepts, or determining what additional data or feedback is needed to address these issues.

To address this challenge, this paper proposes the Diagnosable ColBERT framework, which aligns ColBERT's token embeddings to a reference latent space grounded in clinical knowledge and expert-provided conceptual similarity constraints. This alignment transforms document encodings into inspectable evidence of what the model appears to understand, enabling more direct error diagnosis and more principled data curation without relying on large batteries of diagnostic queries.

The core of Diagnosable ColBERT lies in its diagnostic framework, organized around a pre-existing reference latent space, similar to BioLORD. This latent space needs to accommodate concept names, clinical sentences, and paragraphs, aiming to make contextual token representations clinically legible, not only in terms of term-level concept identity but also in terms of local composition and context-level qualifiers such as negation, temporality, uncertainty, or experiencer.

By mapping late-interaction token representations into a space where these factors can be inspected more directly, Diagnosable ColBERT ensures that retrieval representations remain tied to the diagnosed representation but need not be identical to it. Retrieval embeddings can be learned as a lower-dimensional downprojection of the diagnosed representation, allowing for ranking efficiency without discarding the richer structure needed for diagnosis.

Practical applications of Diagnosable ColBERT include a clinical report retrieval system where testers can issue queries and check if relevant reports are missed, such as when a report only mentions the abbreviation CSD. Diagnosable ColBERT resolves this ambiguity by grounding both sides in a reference latent space, allowing testers to inspect whether query and document representations are correctly positioned near the relevant disease concept, guiding more targeted interventions.