Sentence Splitter: Uncovering Latent Factual Structure for Self-Supervised Learning

The field of NLP has long pursued automatic sentence segmentation and semantic decomposition, historically relying on syntactic parsers, dependency trees, and rule-based systems. Early approaches, such as probabilistic context-free grammars, provided hierarchical span annotations, but required extensive manual labeling. Recent advances with pretrained models like BERT and T5 have improved contextual understanding but still depend on annotated datasets or external tools for structure extraction. Knowledge graph verbalizations and semantic templates have been used to incorporate symbolic knowledge, yet automatic extraction of factual boundaries remains a challenge. Self-supervised learning methods like masked language modeling (MLM) and span corruption have shown success in capturing local context but lack explicit semantic boundary modeling. This work aims to fill the gap by enabling large-scale, label-free factual structure learning through sequence generation, bridging the gap between symbolic knowledge and natural language understanding.

The core challenge is to automatically identify the factual tail segment within a sentence without manual annotations or dependency parsers. Existing methods struggle with the diversity and complexity of natural language, especially in long or syntactically intricate sentences. Precise boundary detection is crucial for constructing accurate knowledge graphs and improving reasoning tasks. The difficulty lies in capturing semantic boundaries that are not always aligned with syntactic cues, and in scaling the process to large corpora. Overcoming these limitations requires a method that can learn semantic segmentation implicitly, leveraging the vast amount of unannotated text, while maintaining high accuracy and robustness across domains.

This work introduces a sequence generation-based approach to sentence splitting, where the model predicts the factual tail directly, bypassing explicit boundary enumeration. It innovates by verbalizing symbolic head-tail pairs into natural language templates, enabling self-supervised training without manual labels. The framework incorporates a lightweight bootstrapping loop, where generated tails are used to expand training data, improving the model iteratively. Unlike prior methods relying on external parsers or span masking, this approach learns semantic boundaries implicitly through probabilistic decoding, making it scalable and domain-agnostic. These innovations collectively enable automatic, structure-aware factual extraction from large unlabeled corpora.

• �� Input: a natural language sentence S, with the goal of identifying the factual tail. • Supervised training: verbalize symbolic head-tail pairs into natural language templates, creating pairs (S, t). • Model training: fine-tune T5 to predict tail t given sentence S, using maximum likelihood estimation, replacing explicit boundary search with probabilistic sequence generation. • Sentence splitting: during inference, generate tail t from S; if t exactly matches a contiguous span, extract it and form prefix p. • Self-augmentation: use the generator to produce alternative tails for p, creating new sentence variants Snew = p + generated tail. • Iterative loop: retrain the splitter on expanded data, further refining boundary detection. • Evaluation: assess accuracy on structured and natural datasets, ensuring generalization.

The experiments utilize ATOMIC2020 and OMCS datasets, with the former providing structured symbolic triples and the latter offering natural language sentences. The models are pretrained with the proposed structure-aware objective and fine-tuned on downstream tasks like CommonsenseQA and knowledge graph completion. Results show the splitter achieves 96% accuracy on ATOMIC verbalizations and 84% on OMCS, outperforming baseline MLM models. The bootstrapping process further improves downstream task performance, with gains of 4-6%. Multiple runs with different seeds confirm stability and robustness. Ablation studies demonstrate the importance of the structure-aware objective and self-augmentation, validating the approach’s effectiveness across diverse scenarios.

The proposed method achieves 96% exact match accuracy on structured ATOMIC data and 84% on natural OMCS sentences, significantly better than traditional span masking. Fine-tuning with structure-aware pretraining boosts CommonsenseQA accuracy by 4 percentage points, from 51.7% to 57.3%, and improves knowledge graph completion ROUGE scores from 20.7% to 27.5%. The lightweight bootstrapping loop stabilizes training, reduces variance, and enhances generalization. These results confirm that learning factual boundaries via sequence generation effectively captures semantic structure, leading to improved reasoning and knowledge extraction performance.

The framework can be applied to automatic knowledge graph construction, enhancing question answering, reasoning, and information retrieval systems. It is especially suited for large-scale, unlabeled corpora, reducing reliance on manual annotation or external parsers. In industry, it can accelerate knowledge base expansion, support domain-specific applications like biomedical or legal NLP, and improve AI interpretability by explicitly modeling factual structures. Long-term, this approach could enable fully autonomous knowledge extraction pipelines, transforming how machines understand and utilize human knowledge in diverse fields.

Current model performance declines with highly complex or lengthy sentences due to increased syntactic and semantic variability. It primarily handles single tail predictions, limiting multi-fact extraction. Generated tails may contain biases or semantic drift, requiring further validation. Computational costs are non-trivial, especially during iterative bootstrapping, and the approach may struggle with highly ambiguous or noisy data. Future work should address multi-tail recognition, robustness, and bias mitigation to realize broader applicability.