SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Bases
Proposes SPARQA, a skeleton-based semantic parser, achieving 21.53 F1 on GraphQuestions and 31.57 P@1 on ComplexWebQuestions, outperforming SOTA.
Key Findings
Methodology
SPARQA employs a high-level skeleton grammar to represent complex questions, combined with BERT-based models for skeleton parsing. The approach involves manual annotation of over 10,000 questions to train the skeleton parser, which performs four subtasks: Split, TextSpanPrediction, HeadwordIdentification, and AttachmentRelationClassification. A multi-strategy scoring mechanism integrates sentence-level pattern matching and word-level similarity, ranking candidate formal queries. Extensive experiments on GraphQuestions and ComplexWebQuestions demonstrate superior performance over existing methods, with ablation studies confirming the importance of skeleton parsing and multi-strategy scoring.
Key Results
- On GraphQuestions, SPARQA achieved an F1 score of 21.53, surpassing PARA4QA's 20.40, a 5.5% improvement. On ComplexWebQuestions, P@1 reached 31.57, outperforming baselines. Skeleton parsing LAS was 93.73, with component accuracies above 97%. Ablation showed removing skeleton parsing decreased P@1 by 6.9%, confirming its critical role. The combined scoring significantly enhanced query matching, demonstrating robustness in complex scenarios.
- The BERT-based skeleton parser supports multi-task fine-tuning, achieving over 97% accuracy in each subtask. The sentence-level scorer, based on pattern matching, contributed most to performance, with a 16% drop when removed. The word-level scorer, using cosine similarity of GloVe embeddings, added further gains. Results indicate that skeleton structure and multi-strategy scoring effectively improve complex question parsing, especially for long-distance dependencies and multi-predicate relations.
- Compared to dependency-based methods, SPARQA's skeleton approach reduces error propagation, leading to better handling of complex, multi-relational questions. The lightweight skeleton formalism simplifies dependency structures, yielding higher accuracy and robustness. These innovations set a new state-of-the-art in KBQA for complex questions, with broad implications for natural language understanding and AI-powered information retrieval.
Significance
This work advances KBQA by addressing the core challenge of parsing complex, long questions with multiple predicates. By introducing a skeleton grammar and leveraging BERT for high-level structure extraction, the approach mitigates dependency parsing errors and enhances interpretability. The multi-strategy scoring further refines query grounding, resulting in significant accuracy gains. Such improvements are crucial for deploying real-world intelligent assistants, search engines, and automated knowledge extraction systems, enabling them to better understand nuanced user queries. The methodology also opens avenues for integrating structured high-level representations into other NLP tasks, fostering more robust and explainable AI systems.
Technical Contribution
The key technical innovation is the skeleton grammar formalism, which abstracts high-level question structure into a directed tree of text spans and attachment relations. This lightweight formalism reduces reliance on dependency parsing accuracy. The BERT-based skeleton parser supports multi-task training for four subtasks, achieving over 97% component accuracy. The multi-strategy scoring combines pattern-based sentence matching with neural similarity measures, effectively ranking candidate formal queries. This integrated framework enhances the robustness and interpretability of semantic parsing, especially for complex questions with multiple predicates and long-distance dependencies.
Novelty
This is the first work to formalize a skeleton grammar specifically for complex question parsing in KBQA, utilizing BERT for high-level structure extraction. Unlike traditional dependency parsers, the skeleton formalism simplifies the structure, reducing error propagation. The multi-strategy scoring mechanism, combining pattern matching and neural similarity, offers a novel way to rank grounded queries. These innovations collectively enable more accurate and robust parsing of complex questions, setting a new benchmark in the field.
Limitations
- The skeleton grammar relies on manual annotation for training data, which is costly and time-consuming, limiting scalability to new domains or languages.
- The approach may struggle with extremely long or nested questions where the high-level skeleton cannot fully capture intricate structures.
- Multi-strategy scoring depends heavily on training data quality; in low-resource settings, performance may degrade. Future work should explore automatic skeleton induction and domain adaptation.
Future Work
Future directions include developing methods for automatic skeleton structure learning to reduce manual effort, integrating multimodal data (images, videos) for richer understanding, and extending the framework to multilingual settings. Improving scalability and generalization across diverse domains remains a priority. Additionally, combining skeleton parsing with other neural architectures, such as graph neural networks, could further enhance the interpretability and accuracy of complex question understanding. These advancements aim to bring robust, scalable, and explainable semantic parsing closer to real-world deployment.
AI Executive Summary
Understanding complex natural language questions remains a significant challenge in knowledge base question answering (KBQA). Traditional dependency parsing methods often falter when faced with long, multi-predicate sentences, leading to errors that cascade through the semantic parsing pipeline. To address this, the authors introduce SPARQA, a novel skeleton-based semantic parsing framework that captures the high-level structure of complex questions. This approach simplifies the dependency structure into a directed tree of text spans connected by attachment relations, effectively reducing parsing errors.
SPARQA leverages a BERT-based model to perform skeleton parsing through four subtasks—Split, TextSpanPrediction, HeadwordIdentification, and AttachmentRelationClassification—each supported by multi-task fine-tuning. The skeleton parser is trained on manually annotated datasets exceeding 10,000 questions, ensuring high accuracy (over 97% in component tasks). The high-level skeleton guides the generation of ungrounded queries, which are then grounded against a knowledge base using structural variants. A multi-strategy scoring mechanism, combining sentence pattern matching and neural similarity, ranks candidate formal queries.
Experimental results on GraphQuestions and ComplexWebQuestions datasets demonstrate that SPARQA surpasses existing state-of-the-art methods, achieving an F1 score of 21.53 and P@1 of 31.57, respectively. Ablation studies confirm the critical role of skeleton parsing and multi-strategy scoring, with significant performance drops observed when these components are removed. The approach not only improves accuracy but also enhances robustness in handling complex, multi-relational questions.
This research marks a significant step forward in semantic parsing for KBQA, offering a scalable, interpretable, and effective solution for understanding intricate natural language queries. Future work aims to automate skeleton learning, incorporate multimodal data, and extend to multilingual scenarios, broadening the impact of this innovative framework.
Deep Dive
Abstract
Semantic parsing transforms a natural language question into a formal query over a knowledge base. Many existing methods rely on syntactic parsing like dependencies. However, the accuracy of producing such expressive formalisms is not satisfying on long complex questions. In this paper, we propose a novel skeleton grammar to represent the high-level structure of a complex question. This dedicated coarse-grained formalism with a BERT-based parsing algorithm helps to improve the accuracy of the downstream fine-grained semantic parsing. Besides, to align the structure of a question with the structure of a knowledge base, our multi-strategy method combines sentence-level and word-level semantics. Our approach shows promising performance on several datasets.