A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems
A prompt-based multi-agent framework raises GPT-4o-mini PrivacyQA performance from 0.394 to 0.587 while narrowing dialect gaps.
Key Findings
Methodology
The framework combines a Dialect Agent and a Privacy Policy Agent. Given dialect information covering phonetic, grammatical, lexical, and cultural properties, the first agent translates a query into Standard American English (SAE) while preserving intent. The policy agent answers from the supplied policy segment and provides a rationale. The dialect agent then audits semantic fidelity; disagreements trigger targeted revision, for at most two iterations. No retraining or dialect-specific fine-tuning is required.
Key Results
- On PrivacyQA, GPT-4o-mini improved from 0.394 zero-shot average F1 to 0.587 with multi-agent zero-shot, and from 0.565 few-shot to 0.598 multi-agent few-shot. Maximum dialect disparity fell from 0.093 to 0.019.
- On PolicyQA, GPT-4o-mini rose from 0.352 zero-shot to 0.452 multi-agent zero-shot, and from 0.449 few-shot to 0.471 multi-agent few-shot. Llama 3.1’s Max Diff dropped from 0.080 to 0.028.
- The approach generalized across models: best AVG Diff reached 0.005 on PrivacyQA and 0.006 on PolicyQA. Adding explicit dialect information raised initial PrivacyQA F1 from 0.5210 to 0.5772.
Significance
The study brings dialect fairness into privacy-policy QA, a high-stakes domain where inaccessible legal language already limits comprehension. Non-dominant dialect speakers may otherwise receive less reliable information about collection, sharing, retention, security, or user rights. The results show that structured prompting can improve both accuracy and equity without collecting sensitive community data for fine-tuning. This offers a practical route toward more inclusive privacy assistants and demonstrates that fairness interventions need not always be parameter-level model changes.
Technical Contribution
The central engineering contribution is a closed-loop protocol: dialect interpretation, evidence-grounded domain answering, dialect-sensitive evaluation, and feedback-based revision. Multi-VALUE’s dialect perturbations are combined with role-specialized prompts, while AVG Diff and Max Diff separately measure average and worst-case inequity. Unlike direct translation or one-shot prompting, the system explicitly checks whether normalization preserved user intent. It works with GPT-4o-mini, Llama 3.1, and DeepSeek-R1, providing a portable, training-free adaptation mechanism.
Novelty
Prior work such as Multi-VALUE, DADA, and TADA primarily evaluates or adapts dialect robustness in general NLP; privacy-policy QA has rarely addressed this dimension. This paper systematically studies multiple dialects in privacy QA and embeds sociolinguistic interpretation inside a domain-QA feedback loop. Its fundamental novelty is a reusable prompt protocol—not a newly trained model—that makes dialect knowledge an explicit computational role.
Limitations
- The evaluation uses Multi-VALUE’s 189 rule-based perturbations, which may not represent real speakers’ regional, age-related, identity-related, or code-switching variation. Consequently, measured fairness may not transfer directly to deployment.
- The pipeline depends on the Dialect Agent’s interpretation. A mistranslation can cause the Privacy Policy Agent to produce a fluent but irrelevant answer, while iterative calls add latency, cost, and opportunities for error propagation.
- The supplied text does not fully report all dialect results, confidence intervals, statistical significance, or comprehensive latency and cost measurements.
Future Work
Future studies should evaluate real community-authored queries across regions and identities, compare automatic normalization with human interpretation, and test additional models, languages, and privacy categories. Deployment research should optimize iteration budgets and cost, require extractive evidence citations, calibrate uncertainty, support human escalation, and audit whether dialect descriptions themselves encode stereotypes.
AI Executive Summary
Privacy policies determine how people understand data collection, sharing, retention, and deletion rights, yet their legalistic language is already difficult to navigate. Large language models add another inequality: systems often answer Standard American English more reliably than non-dominant varieties. In privacy settings, that gap is consequential because communities disproportionately exposed to surveillance and data exploitation may receive the least dependable assistance.
Klisura and colleagues propose a training-free, prompt-based collaboration between two specialized roles. The Dialect Agent receives a compact description of a target dialect and translates the user’s question into SAE while preserving its intent. The Privacy Policy Agent then answers from the relevant policy segment and explains its evidence. The Dialect Agent audits the result against the original wording; if it detects a missed nuance, the policy agent revises the answer, with a maximum of two feedback loops. The design resembles two experts checking both language and legal content rather than one model guessing once.
Results on PrivacyQA and PolicyQA are substantial. GPT-4o-mini’s PrivacyQA average rose from 0.394 zero-shot to 0.587 with the multi-agent zero-shot configuration, while its few-shot score rose from 0.565 to 0.598. On PolicyQA, the corresponding increase was 0.352 to 0.452, and 0.449 to 0.471. PrivacyQA Max Diff fell to 0.019; for Llama 3.1 on PolicyQA it fell from 0.080 to 0.028. The evidence is promising but not conclusive: synthetic dialects, translation errors, extra inference cost, and incomplete reporting of significance remain. Still, the work positions structured agent collaboration as a practical fairness intervention without model retraining.
Deep Analysis
Background
Privacy-language research progressed from OPP-115 practice classification and PolicyIE semantic parsing to PI-Extract entity recognition and specialized QA. PrivacyQA frames answering as sentence selection over mobile-app policies; PolicyQA emphasizes concise span extraction from website policies. In parallel, Multi-VALUE introduced rule-based stress tests across 50 English dialects using 189 perturbation rules. Although dialect disparities are well documented for AAVE and other varieties, domain-specific privacy QA had not been systematically evaluated or mitigated. This paper connects those two research lines.
Core Problem
For dialectal question qd and policy segment p, a QA model f produces an answer evaluated by metric Φ. The paper defines disparity as Δ=max|Φdi(f)-Φdj(f)| across dialects and seeks high average accuracy with low Δ. The challenge is that dialect syntax and vocabulary may differ while legal intent remains constant. Naive normalization can erase pragmatic meaning; direct QA can misread dialectal phrasing. Errors are especially serious when users seek information about rights, sharing, security, or deletion.
Innovation
- ��Role specialization: the Dialect Agent protects linguistic intent; the Privacy Policy Agent supplies policy-grounded content.
- ��Closed-loop verification: the dialect role critiques the answer against the original query and requests revision when needed.
- ��Training-free adaptation: dialect descriptions and prompts replace parameter updates and dialect-specific fine-tuning.
- ��Fairness-oriented evaluation: AVG Diff captures average disparity, while Max Diff exposes the worst cross-dialect gap.
- ��Model portability: the same protocol is tested with GPT-4o-mini, Llama 3.1, and DeepSeek-R1.
Methodology
- ��Input: a policy snippet, a user question in dialect d, and concise dialect information.
- ��Dialect conversion: the Dialect Agent is prompted as a linguistics expert and outputs clear SAE while preserving intent and cultural nuance.
- ��Domain answer: the Privacy Policy Agent receives the normalized query and policy text, then produces a concise answer plus rationale based solely on the segment.
- ��Dialect audit: it sees the original dialectal question, policy, and proposed answer, and checks for misunderstandings.
- ��Revision: negative feedback is sent back to the policy agent; the loop runs no more than two times before acceptance.
- ��Few-shot variant: each agent receives eight examples spanning dialects, question types, and policy scenarios.
- ��Evaluation objective: maximize Φ while minimizing Δ across SAE, RAAVE, Jamaican, Aboriginal, Welsh, and SWE varieties.
Experiments
PrivacyQA contains 1,750 questions, more than 3,500 expert annotations, and 35 mobile-app policies; it is evaluated with answer-classification F1. PolicyQA contains 25,017 question-answer pairs from 115 website policies and uses token-level F1 for span overlap. Multi-VALUE generates dialect variants. Baselines are GPT-4o-mini, Llama 3.1 8B, and DeepSeek-R1-Distill-Qwen-14B in zero-shot and few-shot settings; the proposed framework is tested with and without examples. Ablations compare initial versus final answers and prompts with versus without dialect information.
Results
GPT-4o-mini multi-agent few-shot achieved 0.598 on PrivacyQA versus 0.565 for ordinary few-shot, and 0.471 on PolicyQA versus 0.449. Best PrivacyQA AVG Diff was 0.005 and Max Diff 0.019; PolicyQA AVG Diff reached 0.006. Llama 3.1’s PolicyQA Max Diff declined from 0.080 to 0.028. Explicit dialect information raised PrivacyQA initial average F1 from 0.5210 to 0.5772, indicating that both role collaboration and dialect context matter.
Applications
The framework could power privacy assistants for platforms, governments, and consumer services. Users could ask about third-party sharing, retention, security, access, deletion, or tracking in RAAVE, Jamaican English, or other local varieties. Deployment requires reliable retrieval of the relevant policy segment, a validated dialect description, evidence-linked outputs, monitoring for hallucination, and human escalation. The same architecture may transfer to terms of service, medical consent, insurance, and public-benefit documents.
Limitations & Outlook
The main assumption is that synthetic Multi-VALUE dialects approximate real linguistic diversity; this may fail for identity, context, code-switching, and rapidly changing expressions. Two-agent interaction increases API calls, latency, and cost, and a mistaken dialect translation can propagate into a plausible but wrong policy answer. The paper excerpt does not establish statistical significance, full operational costs, or performance on genuine community data. Future work should add participatory data collection, human audits, calibrated uncertainty, evidence constraints, more languages, and adaptive stopping.
Plain Language Accessible to non-experts
Imagine a public help desk where residents bring a huge, confusing contract. The first clerk is familiar with many communities’ ways of speaking. Before asking what the contract says, this clerk makes sure the resident’s question is understood correctly in a common, easy-to-read form. The second clerk is a contract specialist who is allowed to use only the contract itself. That prevents confident guesses based on general knowledge.
The first clerk then checks the second clerk’s answer against the resident’s original wording. If something was misunderstood, the specialist must revise the answer. This is like having a translator and a lawyer read the same request together, instead of forcing every resident to speak one official style.
In the tests, GPT-4o-mini’s average score rose from 0.394 to 0.587 on PrivacyQA and from 0.352 to 0.452 on PolicyQA. Differences between speaking styles also became smaller. The idea is promising, but the test versions of dialects were generated by rules, so real people must still verify whether the help desk truly understands them.
ELI14 Explained like you're 14
Think of a privacy policy as the ridiculously long rulebook for a social-media game. It says whether the app collects your location, shares your name, keeps your photos, or lets you delete your account. Now imagine asking about the rulebook in the way people in your community naturally speak. A basic bot might understand one style well but get confused by another.
This paper gives the bot two teammates. The Dialect Agent is the language expert: it figures out what the player really means and rewrites the question in Standard American English. The Privacy Policy Agent is the rulebook expert: it searches only the supplied policy text and gives a short answer. Then the language expert checks the answer. If the meaning got lost, the rulebook expert tries again—up to two times.
The scores improved a lot! On PrivacyQA, GPT-4o-mini went from 0.394 to 0.587 in the zero-shot setup. On PolicyQA, it went from 0.352 to 0.452. The gap between dialects shrank too, so the system was not merely better for the already-favored style.
But this is not a perfect cheat code. The dialects were mostly created by transformation rules, not collected from lots of real speakers. The two teammates also require extra model calls, which can cost time and money. Next, researchers should test real conversations and make the bot show the exact policy evidence behind every answer!
Glossary
Dialect Agent
A prompt-defined linguistic specialist that normalizes dialectal questions into SAE and checks whether a response preserves the original intent. It functions as both interpreter and critic.
Used first for query conversion and later for answer validation.
Privacy Policy Agent
A role-prompted domain specialist that answers from a supplied privacy-policy segment. Its instructions require concise, factual, evidence-bounded responses with a rationale.
Generates and revises the policy answer.
Multi-VALUE
A rule-based framework that transforms SAE into synthetic forms of 50 English dialects using 189 linguistically informed perturbation rules. It is used here for stress testing, not retraining.
Creates dialectal versions of PrivacyQA and PolicyQA.
PrivacyQA
A mobile-app privacy-policy dataset with 1,750 questions, over 3,500 expert annotations, and 35 policies. The task classifies candidate sentences as correct or incorrect answers.
Evaluated with answer-classification F1.
PolicyQA
A website privacy-policy dataset containing 25,017 question-answer pairs from 115 policies. It emphasizes extracting short, precise answer spans.
Evaluated with token-level F1.
Max Diff
The largest absolute performance difference between any two dialects. Lower values indicate a smaller worst-case disparity.
Used with AVG Diff to quantify dialect fairness.
Open Questions Unanswered questions from this research
- 1 Do synthetic dialects predict real community performance? Multi-VALUE preserves rule-based semantic content but may miss identity, context, code-switching, and conversational variation; representative user studies are needed.
- 2 What is the operational cost of repeated collaboration? The excerpt does not fully quantify latency, API expense, error propagation, or statistical significance, all of which matter for production systems.
- 3 Can the method transfer beyond English privacy QA? Evidence is still needed for multilingual policies, mixed-language queries, long documents, and adversarial dialect ambiguity.
Applications
Immediate Applications
Multidialect privacy assistant
Platforms or public agencies can pass a user query, dialect description, and retrieved policy segment through the two-agent pipeline. The system can answer questions about sharing, retention, deletion, or access, provided it includes evidence links, uncertainty handling, and human escalation.
Fairness stress testing
Product teams can use Multi-VALUE to generate dialectal test suites, then compare F1, AVG Diff, and Max Diff across SAE, RAAVE, Jamaican, Aboriginal, Welsh, and SWE. Results can identify high-risk policy categories and guide linguistic review.
Long-term Vision
Inclusive legal-information infrastructure
The architecture could extend to terms of service, medical consent, insurance, and public-benefit documents. With participatory data collection, real-user audits, evidence citation, and governance for dialect stereotyping, it could lower access barriers while preserving accountability.
Abstract
Privacy policies inform users about data collection and usage, yet their complexity limits accessibility for diverse populations. Existing Privacy Policy Question Answering (QA) systems exhibit performance disparities across English dialects, disadvantaging speakers of non-standard varieties. We propose a novel multi-agent framework inspired by human-centered design principles to mitigate dialectal biases. Our approach integrates a Dialect Agent, which translates queries into Standard American English (SAE) while preserving dialectal intent, and a Privacy Policy Agent, which refines predictions using domain expertise. Unlike prior approaches, our method does not require retraining or dialect-specific fine-tuning, making it broadly applicable across models and domains. Evaluated on PrivacyQA and PolicyQA, our framework improves GPT-4o-mini's zero-shot accuracy from 0.394 to 0.601 on PrivacyQA and from 0.352 to 0.464 on PolicyQA, surpassing or matching few-shot baselines without additional training data. These results highlight the effectiveness of structured agent collaboration in mitigating dialect biases and underscore the importance of designing NLP systems that account for linguistic diversity to ensure equitable access to privacy information.