Graph-Sparse LDA: A Topic Model with Structured Sparsity
Graph-Sparse LDA uses ontology-guided sparsity, compressing an ASD topic from 119 diagnoses to 6 concepts without losing predictive performance.
Key Findings
Methodology
Graph-Sparse LDA extends LIDA with a three-level generator: documents select topics, topics select latent concept-words, and concept-words generate observed words through an ontology-constrained distribution. Indian Buffet Process priors sparsify the document–topic matrix B and topic–concept matrix A, while P maps concepts to allowable ancestors and descendants. Inference combines blocked Gibbs sampling with Metropolis–Hastings split–merge moves designed to preserve likelihood while increasing sparsity.
Key Results
- On a 31-word binary-tree toy problem, Graph-Sparse LDA achieved slightly higher held-out log likelihood than LIDA across 20 independent instantiations while recovering the three true single-concept topics. It also produced substantially fewer nonzero topic dimensions, showing that ontology information improves interpretability without sacrificing fit.
- On 3,804 autism-spectrum-disorder patients with 3,626 ICD-9-CM diagnoses, predictive performance was on par with or slightly better than LIDA. A representative topic required only 6 concept words to summarize most probability mass, whereas LIDA required 119 diagnoses, capturing links among autism, epilepsy, intellectual disability, and Down syndrome.
- On the MeSH biomedical-literature task, the model again retained test likelihood close to LIDA while producing sparser topics. Samplers ran for 250 iterations; 1% of each dataset was held out, with 20 toy instantiations and 5 independent real-data MCMC runs reported.
Significance
The paper addresses a central interpretability problem in topic modeling: ordinary LDA and sparse variants may still leave hundreds of nonzero words in a topic. Graph-Sparse LDA exploits expert-curated ICD-9-CM and MeSH hierarchies to express a topic through a few human-readable concepts that cover annotation variants. This is valuable for clinical subtype discovery, literature screening, and scientific hypothesis generation, where the topic representation itself matters, not merely downstream prediction.
Technical Contribution
The key technical contribution is separating topic-to-concept structure A from concept-to-word structure P. A can therefore be highly sparse, while each selected concept explains its graph neighborhood. The paper also introduces a joint MH proposal: ontology-based split–merge operations modify A, and a constrained quadratic program constructs P′ so that AP approximately equals A′P′. This keeps likelihood nearly constant and gives sparsity priors more influence. IBP machinery retains nonparametric topic-number learning.
Novelty
The innovation is not simply adding word similarity to LDA; it makes a controlled DAG part of the generative model. Compared with LIDA, the focused topic model, and sparse topic models, Graph-Sparse LDA distinguishes selecting representative concepts from explaining observed terms. Its near-constant-likelihood MH moves specifically address the poor mixing that prevents ordinary Gibbs sampling from deleting topic words in large datasets.
Limitations
- The method assumes that a reliable tree or DAG captures relevant semantic neighborhoods. Missing, incorrect, or cross-branch relations can force probability toward inappropriate concepts and make the resulting summaries misleading.
- The empirical evidence is limited: chains run for 250 iterations, only five real-data MCMC runs were shown, and the paper does not provide extensive convergence diagnostics, runtime scaling, or hyperparameter sensitivity analyses.
- The bag-of-words count model ignores word order, negation, temporal evolution, and richer multimodal clinical signals.
Future Work
Future work should model uncertain or learnable ontologies, combine multiple graphs, and apply analogous MH moves to sparsify B more aggressively. Larger clinical and literature benchmarks should report convergence, computational cost, expert-rated interpretability, and retrieval outcomes. Extensions could add temporal topics, cross-branch relations, contextual language representations, and supervised objectives while preserving graph-based explanations.
AI Executive Summary
Topic models were designed to reveal hidden structure in text, but their outputs often remain difficult to interpret. Standard LDA represents each topic over the entire vocabulary; even sparse models such as LIDA can leave hundreds of nonzero terms. This is especially problematic in biomedicine, where diagnosis codes and MeSH terms already possess expert-defined hierarchies.
Graph-Sparse LDA incorporates that hierarchy into the generative process. A topic first selects a small set of latent concept-words, and each concept explains observed terms among its ancestors and descendants. Indian Buffet Process priors create sparse masks, blocked Gibbs sampling updates the parameters, and ontology-based Metropolis–Hastings split–merge proposals alter A and P while approximately preserving likelihood. Setting P to the identity recovers LIDA.
The experiments show that structure can improve readability without compromising prediction. In a 31-word binary-tree task, the model recovered three underlying concepts and slightly outperformed LIDA on held-out likelihood. In data from 3,804 ASD patients and 3,626 ICD-9-CM diagnoses, one representative topic was summarized by 6 concepts rather than LIDA’s 119 diagnoses, with comparable or slightly better prediction. A MeSH literature task showed the same pattern. The broader lesson is that expert taxonomies can function as statistical explanations, not merely indexing tools. Yet the approach depends on ontology quality, and 250 iterations plus limited repeated runs leave convergence and scalability incompletely established.
Deep Analysis
Background
LDA models documents as mixtures of topics and topics as word distributions. Sparse Topic Model, focused topic models, and LIDA add Bayesian sparsity, but they still treat vocabulary items largely as unrelated dimensions. Biomedical vocabularies are different: ICD-9-CM diagnoses form a hierarchy, and MeSH terms encode nested scientific concepts. Exploiting these structures offers a principled route to shorter, more expert-readable topics.
Core Problem
The objective is to obtain a small set of interpretable concepts without reducing predictive quality. Annotation granularity creates the central difficulty: the same phenomenon may be recorded as a broad parent term or a specific descendant. Independent word sparsity cannot naturally merge such variants. Moreover, with many documents, likelihood dominates the sparsity prior, so standard Gibbs sampling rarely reaches zero counts and mixes slowly toward sparse topic representations.
Innovation
The model introduces a latent concept-word layer between topics and observations. A selects concepts, while P describes which graph-related observed terms each concept can explain. IBP priors retain sparse, potentially unbounded document–topic and topic–concept structures. The distinctive inference contribution is a joint A,P MH proposal: split–merge operations move probability within ontology subgraphs, and constrained optimization finds P′ with AP approximately equal to A′P′. Thus large structural changes can be accepted without major likelihood loss.
Methodology
- �� Generation: IBP-Stick produces topic activity; Bernoulli masks B̄ and Ā determine active document–topic and topic–concept links; B and A use masked Dirichlet distributions.
- �� Assignments: z is sampled from B, latent concept ˜w from A, and observed word w from P˜w. The count-data likelihood is ΣXn˜w log(BnAP˜w).
- �� Graph constraint: P˜w follows Dirichlet(O˜w⊙αP1V), where O permits only ontology ancestors, descendants, and the node itself.
- �� Gibbs updates: intermediate tensors CNKV and CKVV are sampled, followed by B̄, B, Ā, A, and P.
- �� Sparse moves: choose an ontology node, split its mass over descendants or merge descendant mass into the node; construct P⋆ through a constrained quadratic program and sample conservative P′. Accept using the full MH ratio over likelihood, priors, and proposal probabilities.
Experiments
The baseline is LIDA, which subsumes earlier sparse topic approaches. The toy benchmark has 31 vocabulary nodes in a binary tree, three single-concept topics, and 1,000 documents; each concept distributes 10% probability to ancestors and 90% to descendants including itself. Real datasets include 3,804 ASD patients with 3,626 ICD-9-CM diagnoses and a MeSH-annotated biomedical systematic-review corpus. Chains ran for 250 iterations, with 1% held out for predictive log likelihood. Results used 20 toy instantiations and five independent real-data MCMC runs.
Results
Across toy, ASD, and systematic-review settings, predictive performance stayed within a few percentage points of LIDA; the toy and ASD comparisons favored Graph-Sparse LDA slightly. The strongest effect was sparsity: one ASD topic used 6 concept words versus 119 LIDA dimensions while preserving clinically meaningful links among autism, epilepsy, intellectual disability, and Down syndrome. The toy model recovered its three planted concepts, and MeSH experiments showed that compact summaries can coexist with useful retrieval-oriented prediction.
Applications
Clinical researchers can use the model to summarize comorbidity patterns and generate candidate ASD subtypes from longitudinal diagnosis counts. Biomedical information-retrieval teams can use MeSH-based topics to organize systematic-review evidence and prioritize screening. Deployment requires count-based observations and a reasonably accurate tree or DAG; topic summaries should be reviewed by domain experts before clinical use.
Limitations & Outlook
The approach treats the ontology as a strong structural prior, so errors or omissions can distort summaries. It is a static bag-of-words model and does not represent syntax, negation, temporal change, or multimodal measurements. The reported 250 iterations, five real-data runs, and 1% holdout provide useful but limited evidence; runtime, convergence, and human interpretability were not comprehensively evaluated. Future work should learn uncertain relations, support temporal and multi-graph models, and scale inference to larger corpora.
Plain Language Accessible to non-experts
Imagine a hospital with a huge filing cabinet. Each patient record carries labels: some are very specific, such as “intractable focal epilepsy,” while others are broad, such as “epilepsy” or “nervous-system disease.” A basic method treats every label as a separate drawer. A summary can therefore contain more than a hundred labels, making the main pattern hard to see.
Graph-Sparse LDA acts like an archivist who understands the cabinet’s organization. It knows which drawers are parents and which are children, so it can select “epilepsy” as a representative drawer for many nearby specific labels. It first identifies the hidden themes in a record collection, then chooses a few representative concepts for each theme, and finally uses the cabinet’s hierarchy to explain the original labels.
The archivist also tries merges and splits. If many small drawers can be replaced by one broader drawer without making the records much harder to explain, the simpler summary is kept. In the ASD experiment, one topic needed 119 labels under the ordinary sparse method but only 6 concepts here, while prediction remained comparable. The trade-off is dependence on the cabinet design: if the hierarchy is wrong or incomplete, the summary may also be wrong.
ELI14 Explained like you're 14
Imagine sorting your enormous game inventory. You have many versions of swords, potions, armor, and quest items. You want to know your playing style, but listing every item would create a giant, boring spreadsheet.
A normal topic model is like listing every item separately. Graph-Sparse LDA knows the game’s category tree: fire swords are under swords, and swords are under weapons. So instead of naming every item, it can say, “This play style is mostly weapons, fire, and healing.” The category tree is like the medical ontology in the paper.
The model first finds hidden play styles, then chooses a few representative categories, and finally connects specific items to those categories. It also tries combining lots of small categories into one larger one. If the explanation still works, it keeps the shorter version. In the ASD data, one topic shrank from 119 diagnoses to only 6 concepts, while predictions stayed about as good as LIDA’s.
There is a catch! The category tree must be trustworthy. If the game’s labels are badly organized, the model can be misled. It also mostly counts what appears, rather than understanding word order or tone. That makes it especially useful for diagnosis codes and MeSH terms, where experts have already built a hierarchy.
Glossary
Graph-Sparse LDA
A hierarchical topic model that uses a vocabulary graph to produce sparse, interpretable topic summaries. Topics select latent concepts, which then generate observed words.
The paper’s central method, evaluated on ASD diagnoses and MeSH literature.
LIDA
Latent IBP compound Dirichlet Allocation, a nonparametric topic model with IBP-induced sparsity in document–topic and topic–word matrices. It is the principal comparison method.
Graph-Sparse LDA extends LIDA with concept words and ontology-constrained emissions.
Indian Buffet Process
A Bayesian nonparametric prior for sparse binary feature matrices with potentially unbounded latent features. It lets the model learn active topics and connections rather than fixing them in advance.
Used to generate B̄ and Ā sparsity masks.
Concept-word
A latent representative node in the structured vocabulary. It can explain itself and related ancestor or descendant observations.
A models topic-to-concept probabilities, while P models concept-to-word probabilities.
Metropolis–Hastings
A sampling algorithm that proposes a new state and accepts it according to a ratio involving likelihood, priors, and proposal probabilities. Here it enables graph-based sparse structural moves.
Jointly updates A and P to overcome slow Gibbs sparsification.
ICD-9-CM / MeSH
ICD-9-CM is a clinical diagnosis classification system; MeSH is the U.S. National Library of Medicine’s biomedical subject-heading vocabulary. Both contain hierarchical relations.
They provide the graph structures for the two real-world studies.
Open Questions Unanswered questions from this research
- 1 The paper does not systematically quantify how noisy, incomplete, or cross-branch ontologies affect topic quality. Benchmarks with corrupted and multi-relational graphs are needed.
- 2 Convergence, runtime scaling, and behavior beyond 250 iterations remain unclear. Larger patient and literature collections should include diagnostics, wall-clock cost, and sensitivity analyses.
- 3 The static count model cannot explain word order, negation, temporal evolution, or multimodal clinical signals; integrating these while preserving graph-based interpretability remains open.
Applications
Immediate Applications
Clinical subtype discovery
Clinical research groups can input diagnosis counts and the ICD-9-CM hierarchy to obtain compact comorbidity topics. These topics may serve as features for risk or outcome analyses, but experts should validate whether the selected concepts correspond to meaningful patient phenotypes.
Biomedical literature screening
Systematic-review teams can use MeSH-based topics to organize candidate studies and prioritize conceptually relevant records. The model does not replace human eligibility assessment, but it can reduce the burden created by many specific and inconsistently assigned terms.
Long-term Vision
Auditable medical knowledge discovery
With uncertain ontologies, temporal records, and expert feedback, future systems could produce traceable summaries linking each high-level concept to concrete diagnoses or papers. This could support hypothesis generation, evidence synthesis, and controlled knowledge-base updates.
Abstract
Originally designed to model text, topic modeling has become a powerful tool for uncovering latent structure in domains including medicine, finance, and vision. The goals for the model vary depending on the application: in some cases, the discovered topics may be used for prediction or some other downstream task. In other cases, the content of the topic itself may be of intrinsic scientific interest. Unfortunately, even using modern sparse techniques, the discovered topics are often difficult to interpret due to the high dimensionality of the underlying space. To improve topic interpretability, we introduce Graph-Sparse LDA, a hierarchical topic model that leverages knowledge of relationships between words (e.g., as encoded by an ontology). In our model, topics are summarized by a few latent concept-words from the underlying graph that explain the observed words. Graph-Sparse LDA recovers sparse, interpretable summaries on two real-world biomedical datasets while matching state-of-the-art prediction performance.