Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)
A Wizard-of-Oz study of 20 users and 325 questions found text assistants more efficient, while voice and text matched in satisfaction and trust.
Key Findings
Methodology
The authors conducted a Wizard-of-Oz design-probe study with 20 participants reviewing usability-test recordings through simulated text or voice assistants. The interface used React, Socket.IO, and Node.js; the voice condition used the Web Speech API for speech recognition and text-to-speech output. Researchers collected and coded 325 questions, grouped them into information categories, and compared question counts, lengths, and post-study perceptions of efficiency, satisfaction, and trust.
Key Results
- The 325 questions formed five categories: user actions, user mental models, help from the assistant, product and task information, and user demographics. This indicates that evaluators seek not only detected events but also explanations, interpretations, contextual information, and analytic guidance.
- Participants in the text condition asked more questions, but question length did not differ significantly between text and voice. Thus, voice did not automatically produce longer or more elaborated queries, contrary to expectations from some prior conversational-interface studies.
- Text was perceived as significantly more efficient, while satisfaction and trust were rated similarly across modalities. Efficiency therefore appears separable from overall acceptance: voice may feel conversational without improving the speed of professional analysis.
Significance
The paper reframes UX-analysis support from passive visualization toward active, question-driven collaboration. It addresses persistent problems in recording analysis: high cognitive load, limited time and staffing, missed evidence, and evaluator effects. Academically, the dataset of 325 questions provides an empirical map of information needs. Industrially, it suggests that assistants should retrieve evidence on demand, explain their reasoning, and support evaluators’ autonomy rather than merely pushing predetermined analytics.
Technical Contribution
The contribution is primarily interactional and empirical rather than a new recognition algorithm. The authors combine a video player, collapsible chat interface, Socket.IO communication, Web Speech API transcription, and dual text/voice responses into a reusable probe. The resulting framework exposes how evaluators query multimodal recordings and yields concrete engineering requirements: cross-recording synthesis, modality choice, evidence-linked answers, and human oversight. It differs from standard ML visualization systems by making information retrieval responsive to the evaluator’s current question.
Novelty
Relative to commercial features such as UserTesting sentiment analysis and UXTesting emotion detection, and to prior non-interactive ML visualizations, this study systematically characterizes what UX evaluators would ask a conversational assistant during recording analysis. It also directly compares voice and text in this professional context. The fundamental novelty is treating evaluator information needs—not predefined model outputs—as the starting point for assistant design.
Limitations
- Because the assistant was operated through Wizard-of-Oz simulation, the study does not validate automated answer accuracy, latency, hallucination behavior, or error recovery. Real systems may substantially change questioning patterns and trust.
- The sample contained only 20 participants and used one prototype interface and recording context. Expertise, language, culture, task complexity, accessibility needs, and workplace pressure may produce different modality preferences and question distributions.
- The supplied paper excerpt reports significance claims but not full test statistics, effect sizes, or category frequencies, limiting quantitative interpretation of the modality effects.
Future Work
Future systems should integrate video, audio, transcripts, and interaction logs while grounding answers in timestamps and calibrated uncertainty. Follow-up studies should use larger multilingual and cross-industry samples, compare mixed-modality interaction, measure answer correctness and hallucinations, and test how assistants merge analyses from multiple evaluators and recordings without erasing disagreement.
AI Executive Summary
Usability testing reveals where products fail, but analyzing recordings is laborious: evaluators must track screen events, speech, tone, and task progress while taking notes. Existing AI tools, including sentiment or emotion analysis and ML-driven visualizations, generally push predetermined information. They cannot answer the evaluator’s immediate question about a particular moment, inference, or explanation.
Kuang and colleagues addressed this gap at CHI ’23 with a Wizard-of-Oz design probe. Twenty participants reviewed usability recordings using simulated text or voice assistants and asked whatever questions they considered relevant. The researchers collected 325 questions. A React, Socket.IO, and Node.js interface supported text chat; the voice interface used the Web Speech API for transcription and returned both written and spoken responses.
Questions concerned five domains: user actions, user mental models, requests for assistant help, product and task information, and demographics. Text users asked more questions, although question lengths were similar across conditions. Text was judged significantly more efficient, while satisfaction and trust were equivalent. The study’s broader message is that a UX assistant should behave less like a dashboard and more like an accountable analytic colleague: answer on demand, expose evidence, support follow-up questions, and let evaluators choose text, voice, or both. Since responses were simulated and the sample was small, the work is best understood as a requirements map rather than a benchmark of deployed AI accuracy.
Deep Analysis
Background
Usability testing remains a central user-centered method, but manual recording analysis requires simultaneous attention to visual and audio channels. Collaboration improves coverage and reliability, yet only 37% of 279 surveyed UX evaluators reported collaborating on the same recordings. ML classifiers and commercial tools such as UserTesting sentiment analysis and UXTesting emotion detection can surface cues, but prior work found that automated methods do not recover the full set of problems found by human analysis. Non-interactive visualizations also cannot answer targeted questions.
Core Problem
The study asks what evaluators want to ask an assistant during analysis, how question behavior differs between text and voice, and how users perceive each modality. The problem is difficult because UX judgments involve observable actions as well as inferred intentions, mental models, tone, task context, and product knowledge. An assistant must deliver timely information without distracting users or presenting unsupported interpretations as facts.
Innovation
The work introduces conversational questioning as an interaction model for UX recording analysis and uses an open-ended probe to discover needs rather than imposing a fixed task taxonomy. Its innovations are: a five-category account of evaluator questions; a controlled voice-versus-text comparison in the same analysis context; a minimal video-plus-collapsible-chat interface; and multimodal voice responses with optional muting. The contribution is a design and empirical foundation, not a new classifier.
Methodology
- ��Participants: 20 people reviewed usability-test recordings with a simulated assistant.
- ��Text condition: a React chat box accepted typed questions and displayed replies in a chat thread.
- ��Voice condition: participants said “Hey UX assistant” or pressed a microphone button; the Web Speech API transcribed speech, and replies were shown and spoken.
- ��Interaction control: activating the microphone or playing a response automatically paused the video; pulsing halos indicated microphone status, and users could mute speech.
- ��Analysis: researchers coded 325 questions into five categories, compared counts and lengths across modalities, and analyzed survey and interview judgments of efficiency, satisfaction, and trust.
Experiments
This was an empirical design probe rather than a predictive-model benchmark. The two conditions shared the video player, playback controls, and collapsed chat bubble; only the input/output modality differed. Measures included number of questions, question length, content categories, and subjective ratings of efficiency, satisfaction, and trust. No public dataset, training procedure, accuracy metric, or algorithmic baseline was used. Researchers simulated the assistant to isolate information needs before building a fully automated system.
Results
The 325-question corpus showed that evaluators need information ranging from visible actions to hidden mental models and requests for analytic help. Text participants asked more questions, suggesting that typing may better support repeated, low-disruption queries; however, question lengths were not significantly different. Text was perceived as significantly more efficient, while satisfaction and trust did not differ significantly. The findings separate speed from social acceptance and challenge the assumption that voice is inherently more natural or useful for professional analysis.
Applications
UX teams could query a recording about a user’s action, spoken comment, task state, or possible interpretation, then inspect linked evidence. Product teams could connect an assistant to video, transcripts, interaction logs, and demographic metadata to organize findings. Deployment requires timestamp-grounded answers, privacy and access controls, robust multimodal extraction, uncertainty communication, and explicit human review so that hypotheses are not mistaken for observations.
Limitations & Outlook
Wizard-of-Oz operation leaves real-model accuracy, latency, hallucination, and recovery behavior untested. The 20-person sample, single prototype, and specific recording context limit generalizability; speech also depends on microphone permissions, browser support, and noise. Future work should evaluate real assistants in longitudinal industry studies, report complete statistics and effect sizes, compare mixed modalities, test diverse expertise and languages, and design reliable cross-recording synthesis with traceable evidence and preserved disagreement.
Plain Language Accessible to non-experts
Imagine that UX researchers are detectives examining security footage from a store. They must watch where a shopper walks, listen to what the shopper says, notice hesitation, and infer why the shopper chose the wrong door. Traditional AI is like a wall covered with sticky notes: it highlights possible emotions or actions, but the detective cannot ask a note, “Why do you think that?” The detective still has to rewind everything and connect the clues.
This paper tests a different helper: a partner sitting beside the detective. The researcher can type or speak questions such as, “What was the shopper trying to find?” or “Did the instructions confuse them?” In a Wizard-of-Oz study, 20 people used simulated helpers and produced 325 questions. The questions concerned five things: what users did, what they thought, how the helper could assist, what the product and task required, and who the users were.
Typing produced more questions and was judged faster, but people trusted and liked both modes about equally. The lesson is simple: a useful assistant should not dump every possible clue onto the screen. It should wait for the investigator’s question, show where its answer came from, and clearly separate evidence from guesswork.
ELI14 Explained like you're 14
Picture watching a gaming clip to discover why a player keeps getting lost in the same menu. You have to pause, rewind, watch every click, and listen to every comment. Exhausting, right? Now imagine an AI teammate beside you. You could ask, “What was the player looking for?” or “Did that button mean something different to them?”
The researchers did not build a perfect AI immediately. They used a Wizard-of-Oz trick: participants thought they were chatting with a computer, but researchers secretly supplied the replies. Twenty people used either typing or speaking and asked 325 questions. The questions fell into five groups: what the user did, what the user believed, how the AI could help, what the product and task involved, and information about the user.
Here is the surprising part! Typing users asked more questions, but their questions were not longer. They also felt typing was more efficient. Yet voice and text received similar scores for satisfaction and trust. So voice may feel like a friendly conversation without actually making serious analysis faster.
A future UX assistant could be like a super-smart replay partner: it watches the screen, listens to speech, finds important moments, and explains its clues. But it must admit when it is unsure. A confident wrong answer could make a team “fix” a problem that was never really there!
Glossary
Wizard-of-Oz
A study method in which users believe an automated system is operating, while a human secretly simulates its behavior. It allows researchers to study interaction needs before reliable automation exists.
Used to simulate both text and voice assistants without solving full usability-video understanding.
Conversational AI assistant
A system that exchanges natural-language turns and provides information or recommendations in response to user queries. Unlike a static dashboard, it can support follow-up questions.
Proposed as an analytic collaborator for UX evaluators.
UX evaluation
The systematic assessment of how effectively, efficiently, and satisfactorily people can use a product. Usability-test recordings provide behavioral and verbal evidence.
The target workflow in which participants queried the assistant.
User mental model
A user’s internal understanding of how a product works and what its controls or outcomes mean. It is inferred from behavior, language, and hesitation rather than directly observed.
One of the five major question categories.
Web Speech API
Browser interfaces for speech recognition and speech synthesis. Performance depends on the device, browser implementation, permissions, and acoustic environment.
Used for voice-question transcription and spoken responses.
Evaluator effect
The phenomenon that different evaluators identify different usability problems in the same session. Collaboration can improve coverage and reduce dependence on one person’s interpretation.
Motivates treating an assistant as a low-cost analytic partner.
Open Questions Unanswered questions from this research
- 1 A real model’s ability to answer mental-model questions remains unknown. Studies need timestamp-grounded evidence, calibrated confidence, hallucination tests, and human verification.
- 2 The sample cannot establish whether language, culture, expertise, accessibility, or workplace pressure changes question categories or modality preferences; larger cross-context studies are needed.
- 3 Reliable synthesis across multiple evaluators and recordings remains unresolved. Future systems need methods that preserve disagreement, avoid duplicate findings, and expose the provenance of every conclusion.
Applications
Immediate Applications
Conversational recording review
UX evaluators can ask about actions, speech, task progress, and possible causes while watching a test session. The assistant should return the relevant timestamp, source clip, and uncertainty, reducing manual rewind and search. It should support both typing and speech.
Finding organization
Product teams can ask the assistant to cluster observations into user actions, mental models, product context, and other categories, producing a review queue for researchers. Integration should include transcripts and logs, access control, privacy protection, and human confirmation.
Long-term Vision
Multi-evaluator analytic workspace
A future assistant could combine several evaluators, recordings, transcripts, interaction logs, and acoustic cues, then highlight agreement and disagreement with traceable evidence. This could lower collaboration costs, but bias, privacy, provenance, and error propagation must be controlled.
Abstract
AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.