Introducing Axlerod: An LLM-based Chatbot for Assisting Independent Insurance Agents

The evolution of AI in insurance has transitioned from rule-based automation to predictive analytics, machine learning, and now large language models (LLMs). Early systems like IntelliBot demonstrated basic inquiry handling, but lacked contextual understanding and dynamic data access. Recent advances, such as GPT-3 and Llama 3, have shown remarkable text comprehension, prompting industry interest in integrating LLMs for risk assessment, claims processing, and customer engagement. Major software providers like Guidewire are embedding LLMs into core platforms, aiming to improve decision-making and operational efficiency. Despite these developments, most solutions remain customer-facing, with limited focus on supporting internal workflows of insurance agents, who handle complex policy data and require high accuracy and reliability. Data privacy, model transparency, and real-time access remain key challenges for broader adoption.

Insurance agents face significant inefficiencies due to reliance on manual searches within large, complex policy databases. Traditional search tools are slow and often produce ambiguous results, leading to delays and errors. Existing AI solutions lack industry-specific customization, struggle with dynamic data, and cannot handle multi-turn, context-aware queries effectively. These limitations hinder agents' ability to provide timely, accurate service, impacting customer satisfaction and operational costs. The core problem is designing an AI assistant capable of understanding nuanced policy language, accessing real-time data, and supporting multi-step interactions—requirements that exceed the capabilities of generic chatbots. Addressing these issues is critical for enabling scalable, reliable agent support in high-stakes, regulated environments like insurance.

This work introduces a novel integration of Google Gemini 2.5 Pro with a retrieval-augmented generation framework tailored for insurance workflows. Key innovations include: 1) Combining LLMs with structured tools—policy detail, search, and documentation retrieval—to enable dynamic, real-time data access; 2) Developing Smoltalk, a microframework that simplifies tool orchestration and API integration; 3) Fine-tuning the model with industry-specific insurance knowledge bases, improving understanding of policy language; 4) Supporting multi-turn, context-aware dialogues for complex queries. These innovations address the limitations of existing models, enabling accurate, fast, and reliable internal support for insurance agents, a previously underexplored area in AI research.

• �� Core model selection: Google Gemini 2.5 Pro, fine-tuned with insurance domain data. • Tool design: policy detail retrieval, policy search via Typesense, documentation search. • Microframework Smoltalk: encapsulates tool calls, manages API interactions, simplifies integration. • Intent recognition: employs deep learning classifiers trained on insurance-specific datasets to understand user queries. • Context management: maintains dialogue history for multi-turn interactions. • Data access: connects to a large insurance policy database (~730,000 policies), claims, and documentation (~400MB). • Response generation: combines retrieved data with LLM outputs, applying prompt engineering to ensure accuracy. • Evaluation: accuracy metrics, response time measurement, cost analysis, and robustness testing across multiple query types.

The system was tested in a controlled environment simulating typical agent workflows, including policy lookup, coverage verification, and document retrieval. A dataset of active policies (~730,000 entries) was used for validation. Baseline comparisons involved manual searches and traditional keyword-based systems. Metrics included accuracy (overall 93.18%), response time (average 2.42 seconds faster), and cost per query (~$0.0075). Multiple question variants tested the model’s understanding of different phrasings. Automated Python unittest scripts verified response consistency. Results confirmed the system’s robustness, especially in multi-turn dialogues and complex queries, demonstrating its practical utility for real-world deployment.

Axlerod outperformed traditional methods with a 93.18% accuracy across tasks like policy ID, auto-pay eligibility, vehicle coverage, and billing plans. It reduced average search time by 2.42 seconds, with significant improvements in multi-step and cross-page queries. Cost per response was minimal, supporting large-scale adoption. The system maintained high performance even with ambiguous or complex queries, validating its reliability. These results suggest that integrating LLMs with structured insurance data can revolutionize internal workflows, reducing agent workload and improving service speed and accuracy.

The system can be integrated into existing agent platforms to facilitate rapid policy retrieval, documentation access, and customer query handling. It requires access to insurance databases, API setup, and agent training. In the short term, it enhances routine tasks, reducing search times and errors. Long-term, it can evolve into a comprehensive decision support tool, assisting in claims processing, underwriting, and risk management, ultimately transforming insurance operations into more efficient, data-driven processes.

Reliance on database quality and real-time updates limits response accuracy; outdated data can cause errors. Handling ambiguous queries remains challenging, requiring further refinement of intent recognition. Deployment at scale involves addressing multi-user concurrency, data privacy, and security concerns, which are not fully resolved in current prototypes. Future improvements should focus on robustness, scalability, and privacy safeguards to ensure safe, reliable, and compliant deployment.