WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
WearableQA benchmark evaluates AI's health reasoning on real wearable data, with performance ranging from 19.6% to 72.9%.
Key Findings
Methodology
Utilizes a dual-grounding framework combining literature-grounded physiological findings with statistically validated population-grounded patterns to construct reliable questions. Evaluates 14 proprietary and open-source LLMs, effectively distinguishing model capabilities.
Key Results
- Result 1: Gemini-3.1-Pro model achieved the highest accuracy of 72.9% on WearableQA, significantly above the 10% random baseline.
- Result 2: Among open-source models, Gemma-4-26B-A4B performed best with an accuracy of 42.5%.
- Result 3: Most models struggled with data reasoning, especially in cross-signal reasoning.
Significance
WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data. It addresses the reliance on synthetic signals in existing benchmarks, promoting advances in personalized health understanding and monitoring.
Technical Contribution
WearableQA offers fine-grained diagnosis of model reasoning capabilities through a dual-grounding framework and a diagnostic taxonomy of 16 question types. It combines literature and population-grounded physiological patterns to ensure reliable and non-trivial questions.
Novelty
WearableQA is the first benchmark to evaluate data and health reasoning on real-user longitudinal wearable records, integrating blood biomarkers and demographic information.
Limitations
- Limitation 1: Most models perform poorly in data reasoning when deriving answers directly from raw measurements.
- Limitation 2: Cross-signal reasoning remains challenging, with stronger models showing clear drops in performance.
Future Work
Future directions include improving model performance in data and cross-signal reasoning, and expanding the benchmark to cover more physiological signals and health metrics.
AI Executive Summary
Advances in wearable sensing have enabled continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate AI systems' ability to reason over real users' longitudinal wearable records. WearableQA benchmark evaluates models' capabilities in data and health reasoning through 4,084 multiple-choice questions, covering long-term measurements from 200 real users. Evaluation results show significant variation in model performance, with accuracy ranging from 19.6% to 72.9%. WearableQA not only provides fine-grained diagnosis of model capabilities but also reveals shortcomings in data and cross-signal reasoning. Future research directions include improving model performance in these areas and expanding the benchmark to cover more physiological signals and health metrics.
Deep Analysis
Background
Recent advancements in wearable sensing technologies have enabled continuous monitoring of physiological and behavioral signals. These signals are used to support personalized health understanding and lifestyle assessment. However, existing benchmarks largely rely on synthetic or simulated signals, lacking evaluation of reasoning capabilities on real users' longitudinal wearable records.
Core Problem
The core problem is whether existing LLMs can reason over real users' longitudinal wearable records, particularly in data and health reasoning. This issue is crucial as LLM-based health assistants emerge, requiring models to extract information from noisy physiological measurements and interpret it in a health-related context.
Innovation
The core innovation of WearableQA lies in its dual-grounding framework, combining literature-grounded physiological findings with statistically validated population-grounded patterns. This approach ensures reliable and non-trivial questions and provides fine-grained diagnosis of model reasoning capabilities.
Methodology
- �� Utilizes a dual-grounding framework combining literature-grounded physiological findings and statistically validated population-grounded patterns.
- �� Constructs 4,084 multiple-choice questions covering long-term measurements from 200 real users.
- �� Evaluates 14 proprietary and open-source LLMs, analyzing their capabilities in data and health reasoning.
Experiments
The experimental design includes evaluating 14 LLMs using the 4,084 multiple-choice questions in the WearableQA benchmark. Each model is evaluated under the same protocol, with questions including wearable measurements, demographic information, and blood biomarkers.
Results
Experimental results show significant variation in model performance, with accuracy ranging from 19.6% to 72.9%. Gemini-3.1-Pro model performed best, while among open-source models, Gemma-4-26B-A4B was the top performer.
Applications
WearableQA can be used to evaluate and improve LLMs' applications in personalized health understanding and monitoring. It provides a realistic and diagnostic benchmark for models' reasoning capabilities on real-world wearable data.
Limitations & Outlook
While WearableQA offers a comprehensive evaluation framework, most models still perform poorly in data and cross-signal reasoning. Future research needs to improve these areas and expand the benchmark to cover more physiological signals and health metrics.
Plain Language Accessible to non-experts
Imagine you have a smart watch that records your daily activities. WearableQA is like a teacher giving tests on the data collected by these watches to see if AI can understand what the data means. For example, it might ask, "What does it mean if someone's heart rate has been high for the past month?" The AI needs to find the answer from the data, just like a student finds answers from a textbook. This process helps us understand AI's ability to handle real-world data and its potential in health monitoring.
ELI14 Explained like you're 14
Imagine you have a super smart watch that tracks your daily activities, like how many steps you took and how fast your heart is beating. WearableQA is like a test to see if AI can figure out your health status from this data. For example, it might ask, "What does it mean if someone's heart rate has been high for the past month?" The AI needs to find the answer from the data, just like you find answers in a test. This helps us understand AI's ability to handle real-world data and its potential in health monitoring.
Glossary
Longitudinal Time Series
A sequence of data collected over a long period, often used to analyze trends and patterns.
Used in the paper to evaluate AI's reasoning on long-term wearable records.
Cross-Signal Integration
Integrating data from multiple signal sources to provide a comprehensive analysis.
Used in the paper to evaluate AI's reasoning capabilities in integrating multiple physiological signals.
Biomarker
A measurable biological indicator used to assess health status.
Used in the paper to combine with wearable data for health reasoning.
Dual-Grounding Framework
Combines literature-grounded and population-grounded physiological patterns to ensure reliable questions.
Used in the paper to construct reliable reasoning questions.
Diagnostic Benchmark
A benchmark used to evaluate and diagnose model capabilities.
Used in the paper to evaluate LLM reasoning on real-world wearable data.
Open Questions Unanswered questions from this research
- 1 How to improve model performance in cross-signal reasoning, as current methods still fall short.
- 2 Existing models perform poorly in data reasoning when deriving answers directly from raw measurements, requiring further research.
Applications
Immediate Applications
Personalized Health Monitoring
By evaluating AI's reasoning on wearable data, it helps develop smarter health monitoring systems.
Long-term Vision
Intelligent Health Assistant
Develop AI assistants capable of extracting health information from long-term wearable data, providing personalized health advice.
Abstract
Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whether AI systems can reason over a real user's longitudinal wearable record. We introduce WearableQA, a benchmark comprising 4,084 10-option multiple-choice questions constructed from the wearable time series, blood biomarkers, and demographics of 200 real users, each with up to 500 days of daily measurements. WearableQA preserves authentic wearable distributions that include device noise and inter-individual variability. To evaluate distinct reasoning capabilities, we introduce 16 question types organized along two complementary axes: data versus health reasoning, which distinguishes computation over longitudinal measurements from physiological interpretation; and single- versus cross-signal reasoning, which separates reasoning about individual signals from the integration of multiple signals. To construct reliable questions at scale, we adopt a dual-grounding framework that combines literature-grounded physiological findings with statistically validated population-grounded physiological patterns. This enables the capture of meaningful relationships observed in real-world wearable data. Evaluation of 14 proprietary and open-source LLMs demonstrates that WearableQA effectively differentiates model capabilities, with performance ranging from 19.6% to 72.9% against a 10% chance baseline. Moreover, WearableQA remains far from solved: most models achieve accuracies below 60%. Overall, WearableQA provides a realistic and diagnostic benchmark for evaluating LLM reasoning over real-world wearable data.