Bridging the Interpretation Gap in Accessibility Testing: Empathetic and Legal-Aware Bug Report Generation via Large Language Models
HEAR transforms accessibility logs into empathetic, legal-aware reports using semantic slicing and multi-layer reasoning.
Key Findings
Methodology
HEAR consists of three phases: context reconstruction via semantic slicing, dynamic persona injection, and multi-layer causal reasoning. It combines JSON logs, UI screenshots, and legal standards to generate stakeholder-friendly reports.
Key Results
- Result 1: HEAR achieved 100% accuracy across 103 issues in visual grounding, textual fidelity, and logical reasoning.
- Result 2: User studies showed HEAR reports improved empathy, urgency, and legal risk awareness by ~40% over raw logs.
- Result 3: HEAR was preferred by 100% of participants for communicating issues to non-technical stakeholders.
Significance
HEAR addresses the interpretation gap in accessibility testing by translating technical logs into human-centric narratives. It improves issue prioritization and fosters awareness of user impact and legal risks.
Technical Contribution
HEAR introduces dynamic persona injection combined with multi-layer reasoning, enhancing report readability and persuasiveness. Its semantic slicing optimizes LLM processing efficiency by reducing noise.
Novelty
HEAR is the first framework to directly link accessibility issues with legal standards, using dynamic personas and causal reasoning to bridge technical logs and user experience gaps.
Limitations
- Limitation 1: HEAR relies on the accuracy of upstream detection tools; errors propagate downstream.
- Limitation 2: Legal analysis is region-specific, requiring database expansion for global applications.
- Limitation 3: Computational cost may limit real-time applications.
Future Work
Future work includes expanding legal databases for broader coverage, optimizing computational efficiency for real-time reporting, and exploring applications in web accessibility testing.
AI Executive Summary
Modern accessibility testing tools detect interface violations but produce technical outputs that non-specialists struggle to interpret. HEAR addresses this gap by transforming raw logs into empathetic, legal-aware reports using semantic slicing, dynamic persona injection, and multi-layer reasoning.
In experiments with 103 issues across four popular Android apps, HEAR demonstrated high accuracy and significantly improved stakeholder understanding of user impact and legal risks. User studies confirmed its superiority over raw logs in empathy, urgency, and persuasiveness.
While HEAR depends on upstream tool accuracy, its innovative approach provides a new solution for accessibility reporting, paving the way for future research and broader applications.
Deep Analysis
Background
Tools like Google Accessibility Scanner and Groundhog have lowered the barrier for detecting interface violations. However, their technical outputs lack semantic context, making them inaccessible to non-technical stakeholders and hindering remediation rates.
Core Problem
The core issue is the interpretation gap: technical logs fail to convey user impact and legal risks effectively. This prevents teams from prioritizing fixes and understanding the broader implications of accessibility issues.
Innovation
HEAR bridges this gap by combining dynamic persona injection and causal reasoning to link accessibility issues with user experience and legal standards. Its semantic slicing reduces noise and enhances LLM efficiency.
Methodology
- �� Context reconstruction: Semantic slicing extracts localized information and aligns it with screenshots for visual grounding.
- �� Dynamic persona injection: Selects personas based on violation type to simulate user interaction.
- �� Multi-layer reasoning: Uses CoT to analyze physical barriers, functional disruptions, and legal risks.
Experiments
Experiments involved 103 issues from Instagram, Wish, and two other apps, covering touch target size, content labeling, and contrast ratio. Reports were generated using GPT-4 and evaluated via user studies.
Results
HEAR reports achieved 100% accuracy in grounding and reasoning. User studies showed significant improvements in empathy, urgency, and legal risk awareness compared to raw logs.
Applications
HEAR can be used in mobile app development to prioritize accessibility fixes and enhance legal compliance awareness among teams.
Limitations & Outlook
HEAR relies on detection tool accuracy, has region-limited legal analysis, and incurs high computational costs. Future work should address these challenges.
Plain Language Accessible to non-experts
Imagine using a shopping app where buttons are too small to tap, making checkout impossible. For someone with Parkinson's, this isn't just annoying—it's a complete barrier. HEAR acts like a translator, turning technical issues into user stories that developers can understand, highlighting the problem's impact and legal risks.
ELI14 Explained like you're 14
Ever tried playing a game where the buttons were too tiny to press? Frustrating, right? Now imagine someone with shaky hands—they can't press those buttons at all! HEAR is like a super helper that tells developers how bad the problem is and even warns them about breaking laws. Cool, huh?
Glossary
Semantic Slicing
Extracts localized semantic data to reduce noise.
Used to process UI elements efficiently.
Dynamic Persona
Simulates user profiles based on issue type.
Generates empathetic narratives.
Causal Reasoning
Analyzes physical, functional, and legal impacts step-by-step.
Explains multi-layer consequences of accessibility issues.
Touch Target Size
Minimum size for interactive elements, typically 48dp.
Evaluates button accessibility.
Legal Compliance
Ensures software adheres to regional accessibility laws.
Assesses legal risks of violations.
Open Questions Unanswered questions from this research
- 1 How can HEAR support global legal standards?
- 2 How can HEAR's computational efficiency be improved for real-time use?
Applications
Immediate Applications
Mobile App Accessibility Fixes
Developers can use HEAR reports to quickly identify and prioritize fixes.
Legal Risk Assessment
Product managers can use HEAR to evaluate potential legal risks and adjust strategies.
Long-term Vision
Cross-Platform Accessibility Testing
Expand HEAR to web and desktop applications, advancing global accessibility standards.
Abstract
Modern automated accessibility testing tools for mobile applications have significantly improved the detection of interface violations, yet their impact on remediation remains limited. A key reason is that existing tools typically produce low-level, technical outputs that are difficult for non-specialist stakeholders, such as product managers and designers, to interpret in terms of real user harm and compliance risk. In this paper, we present \textsc{HEAR} (\underline{H}uman-c\underline{E}ntered \underline{A}ccessibility \underline{R}eporting), a framework that bridges this interpretation gap by transforming raw accessibility bug reports into empathetic, stakeholder-oriented narratives. Given the outputs of the existing accessibility testing tool, \textsc{HEAR} first reconstructs the UI context through semantic slicing and visual grounding, then dynamically injects disability-oriented personas matched to each violation type, and finally performs multi-layer reasoning to explain the physical barrier, functional blockage, and relevant legal or compliance concerns. We evaluate the framework on real-world accessibility issues collected from four popular Android applications and conduct a user study (N=12). The results show that \textsc{HEAR} generates factually grounded reports and substantially improves perceived empathy, urgency, persuasiveness, and awareness of legal risk compared with raw technical logs, while imposing little additional cognitive burden.