MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

Recent progress in large language models (LLMs) like GPT-4, Llama 2, and Vicuna has significantly advanced NLP capabilities, especially in question answering, summarization, and dialogue. However, these models still struggle with multi-step reasoning tasks that require integrating multiple facts, commonsense, and long-term context. Existing datasets such as RuleTakers, EntailmentBank, and SocialIQA have contributed to understanding reasoning, but they often lack the complexity and realism needed to evaluate models in real-world scenarios. Many prior datasets are artificially constructed, with limited scope for multi-layered inference or natural language richness. Consequently, there is a pressing need for a comprehensive, scalable benchmark that combines natural language narratives with complex reasoning chains, capturing the nuances of human cognition and commonsense reasoning. MuSR addresses this gap by leveraging neurosymbolic algorithms to generate high-fidelity, multi-domain datasets that challenge current models and foster future innovations.

Despite significant advances, current AI models exhibit notable deficiencies in robust, multi-step reasoning within realistic contexts. The core challenge lies in creating datasets that are both sufficiently complex and natural, enabling models to learn and generalize reasoning skills applicable to real-world tasks. Existing benchmarks either oversimplify reasoning processes or rely on artificial, rule-based data that do not reflect the intricacies of human cognition. As a result, models often fail to maintain fact consistency, handle long narratives, or perform multi-layered inference reliably. Addressing these issues requires a new approach to data generation—one that can produce realistic, diverse, and challenging reasoning scenarios at scale, providing a meaningful testbed for model evaluation and development.

MuSR introduces a pioneering approach combining neurosymbolic inference trees with automated natural language story generation. Key innovations include: • A structured reasoning tree construction algorithm that recursively expands facts, ensuring multi-step inference complexity; • Long-form narrative synthesis guided by these trees, maintaining factual coherence and narrative realism; • Multi-stage validation using GPT-4 to verify fact entailment and reasoning consistency. This integrated pipeline allows scalable, high-quality data generation, surpassing prior datasets that are either artificial or lack depth. The approach enables continuous expansion, domain diversity, and realistic storytelling, setting new standards for reasoning benchmarks.

• �� Tree template construction: Define initial facts and reasoning strategies for each domain, creating a high-level inference template.
• �� Reasoning tree completion: Recursively sample supporting facts from GPT-4, expanding each node to form multi-layered inference chains, with validation filters to prevent shortcuts.
• �� Story generation: Segment the reasoning tree’s facts into chunks, prompting GPT-4 to generate narrative chapters that embed these facts naturally.
• �� Fact validation: Use GPT-4 to verify the factual and logical consistency of the generated stories, ensuring high-quality data.
• �� Final story assembly: Concatenate chapters, refine narratives, and prepare datasets for evaluation.

The dataset comprises 756 instances across three domains, with an average of 10+ reasoning steps per story. Models like GPT-4, Llama 2, and Vicuna were tested using chain-of-thought prompting, revealing accuracy around 50%, significantly below human performance (~95%). Ablation studies showed that deeper reasoning trees and richer facts improved model accuracy, confirming the importance of complex inference structures. The experiments validated MuSR’s difficulty and its potential to benchmark future reasoning models. Human annotators rated the stories as highly realistic and consistent, supporting the dataset’s quality and relevance.

Models achieved approximately 50% accuracy, highlighting the challenge of multi-step soft reasoning. Chain-of-thought prompting improved performance but left significant gaps. Deeper reasoning trees and increased fact diversity correlated with higher accuracy, validating the dataset’s design. Human performance was near 95%, establishing a high benchmark. The rule-based baselines performed near chance, confirming the dataset’s difficulty. These results underscore the need for more advanced reasoning algorithms and training strategies.

MuSR can serve as a rigorous benchmark for developing and evaluating reasoning capabilities in NLP systems, especially in domains requiring complex inference like legal analysis, medical diagnosis, and investigative journalism. Its multi-domain design allows for targeted training and assessment of models’ reasoning depth, factual consistency, and narrative understanding. Additionally, it can facilitate research into interpretability and robustness of AI reasoning processes, ultimately contributing to more reliable and explainable AI systems.

The dataset relies heavily on GPT-4 for generation, which may introduce biases or limit diversity. The models tested still struggle with very long, intricate narratives, indicating room for improvement in memory and inference mechanisms. Current focus on specific domains restricts generalization to broader reasoning tasks. Future work should incorporate multi-modal data, expand reasoning types, and explore less dependency on a single generative model to enhance dataset robustness and applicability.