Satyrn: A Platform for Analytics Augmented Generation
Satyrn combines structured data analysis with LLMs, achieving 86% factual accuracy in report generation.
Key Findings
Methodology
Satyrn employs Analytics Augmented Generation (AAG), leveraging structured data analysis to produce fact sets that guide large language models (LLMs). The system defines knowledge rings (rings) to abstract data objects, uses Structured Question Representation (SQR) for complex analysis plans, and features a deterministic analysis engine converting plans into SQL queries. Facts derived are expressed as natural language statements, which are then used to prompt LLMs. This approach ensures high factual accuracy, with experiments showing over 86% correctness across multiple domains, outperforming GPT-4 Code Interpreter at 57%.
Key Results
- In a set of 200 reports, Satyrn achieved over 86% factual claim accuracy, significantly surpassing baselines, especially when using smaller models like Mistral-7B, which reached 86% accuracy compared to 57% for GPT-4 Code Interpreter.
- Reports maintained high fluency (scores ~0.944) and coherence (~0.977), demonstrating that fact-driven prompts do not compromise language quality.
- Ablation studies confirmed that structured natural language facts outperform table-based inputs, especially for small models, validating the effectiveness of the fact expression approach.
Significance
This work addresses the longstanding challenge of factual reliability in AI-generated reports. By integrating deterministic structured analysis with neural language models, Satyrn offers a scalable, domain-agnostic framework capable of producing trustworthy, coherent reports grounded in large-scale databases. Its ability to operate effectively with smaller models broadens accessibility, making it highly impactful for automated reporting, decision support, and knowledge management in data-rich but text-sparse environments. The approach paves the way for more reliable AI systems in critical applications, reducing misinformation and increasing user trust.
Technical Contribution
The paper introduces a novel neuro-symbolic architecture combining knowledge rings, SQR plans, and a deterministic analysis engine to ensure factual correctness. It separates analysis from language generation, enabling explicit control over data retrieval and reasoning. The system supports multi-model deployment, with fact expression optimized for small models, and offers a flexible, reusable framework for various report types across domains. These innovations collectively enhance the reliability, interpretability, and scalability of AI-generated reports.
Novelty
This is the first integration of structured data analysis with LLMs via a formal, deterministic analysis plan (SQR) and knowledge rings, enabling factually grounded natural language reports. Unlike prior methods like RAG or code interpreters, Satyrn guarantees analysis correctness and supports multi-model deployment, representing a significant leap in trustworthy AI report generation.
Limitations
- The system relies on predefined knowledge rings and analysis templates, requiring manual effort to extend to new domains or data sources, limiting full automation.
- Complex analyses and large datasets increase computational costs, impacting real-time deployment capabilities.
- Dependence on structured data restricts handling of unstructured or multimodal information, necessitating future integration with other data modalities.
Future Work
Future directions include automating knowledge ring creation, developing adaptive analysis plans, and integrating multimodal data sources. Enhancing real-time processing and expanding domain coverage will further increase system versatility. Additionally, exploring unsupervised or semi-supervised methods for dynamic template generation could reduce manual effort, broadening applicability across industries such as healthcare, finance, and urban planning.
AI Executive Summary
The rapid advancement of large language models (LLMs) has revolutionized natural language generation, yet their tendency to produce factually inaccurate content remains a critical obstacle, especially in professional and decision-making contexts. Traditional retrieval-augmented methods like RAG leverage external textual sources to improve accuracy but are limited by their dependence on textual retrieval, which cannot fully exploit structured data like relational databases. Addressing this gap, Satyrn introduces a neuro-symbolic framework that integrates structured data analysis with LLMs, ensuring factual correctness and coherence.
At its core, Satyrn employs a knowledge ring (ring) abstraction to represent data objects, attributes, and relationships in a domain-agnostic manner. It uses a formalized analysis plan language, SQR, to specify complex analytical workflows, which are then executed by a deterministic analysis engine translating plans into SQL queries. The resulting facts are expressed as natural language statements, forming a reliable factual foundation for report generation. These facts are incorporated into prompts for LLMs, guiding the generation process without sacrificing fluency.
Experimental results demonstrate that Satyrn achieves over 86% factual accuracy across diverse domains, outperforming baselines like GPT-4 Code Interpreter at 57%. Notably, even smaller models such as Mistral-7B reach high accuracy levels (~86%), highlighting the efficiency of structured fact expression. The reports maintain high fluency and coherence, verified through human and automatic evaluations.
This approach significantly advances trustworthy AI, offering a scalable, interpretable, and domain-agnostic solution for automated report generation. Its ability to operate effectively with smaller models broadens accessibility and deployment potential. Future work will focus on automating knowledge ring creation, expanding multimodal capabilities, and optimizing real-time analysis, aiming to embed this technology into real-world applications like healthcare, urban planning, and financial analysis. Despite current limitations in automation and computational costs, Satyrn sets a new standard for factually reliable AI-generated content, promising a future where AI can support critical decision-making with unprecedented trustworthiness.
Deep Dive
Abstract
Large language models (LLMs) are capable of producing documents, and retrieval augmented generation (RAG) has shown itself to be a powerful method for improving accuracy without sacrificing fluency. However, not all information can be retrieved from text. We propose an approach that uses the analysis of structured data to generate fact sets that are used to guide generation in much the same way that retrieved documents are used in RAG. This analytics augmented generation (AAG) approach supports the ability to utilize standard analytic techniques to generate facts that are then converted to text and passed to an LLM. We present a neurosymbolic platform, Satyrn, that leverages AAG to produce accurate, fluent, and coherent reports grounded in large scale databases. In our experiments, we find that Satyrn generates reports in which over 86% of claims are accurate while maintaining high levels of fluency and coherence, even when using smaller language models such as Mistral-7B, as compared to GPT-4 Code Interpreter in which just 57% of claims are accurate.