Interactive Multi-Turn Retrieval for Health Videos
Proposed DATR framework with multi-turn query fusion significantly improves health video retrieval, achieving R@1 of 19.5% on MHVRC.
Key Findings
Methodology
This work constructs the MHVRC corpus by pairing health instructional videos with detailed video-grounded descriptions and refined user queries generated via VideoChat-Flash and DeepSeek. The DATR framework employs a two-stage retrieval process: Stage I uses a CLIP-style dual encoder with sparse frame sampling for efficient coarse retrieval, mapping text and videos into a shared embedding space. Stage II fuses multi-turn queries through a neural fusion module (MLP) and re-ranks the top candidates using a lightweight cross-encoder that captures fine-grained procedural semantics. The training optimizes contrastive loss for the encoders, ensuring semantic alignment. This approach balances retrieval speed and precision, effectively modeling user intent evolution in health scenarios.
Key Results
- On the MHVR dataset, DATR achieves R@1 of 19.5%, outperforming the strongest single-turn baseline HERO at 15.2%, with a 4.3% improvement. R@10 reaches 57.1%, and median rank drops from 37.2 to 25.6, indicating more accurate retrieval. The model also shows consistent gains across recall metrics, demonstrating its effectiveness in capturing procedural details.
- User studies confirm that multi-turn queries better express fine-grained procedural semantics, leading to more clinically relevant results. Ablation experiments reveal that query fusion and lightweight cross-encoding contribute significantly to performance improvements.
- The results validate that integrating dialogue-aware multi-turn modeling enhances health video retrieval, especially in complex, safety-critical contexts where detailed procedural understanding is essential.
Significance
This research addresses a critical gap in health AI by enabling interactive, multi-turn retrieval that aligns with real-world clinical and educational workflows. The constructed corpus and proposed framework set new benchmarks for dialogue-based multimodal retrieval, facilitating personalized health information access. The approach improves the precision and relevance of retrieved videos, supporting applications like remote medical training, patient education, and rehabilitation guidance. It also opens avenues for developing more intelligent health assistants capable of understanding evolving user needs, thus advancing the field toward more human-centric AI systems.
Technical Contribution
The paper introduces a novel two-stage retrieval framework, DATR, combining the efficiency of dual encoders with the fine-grained understanding of lightweight cross-encoders. It innovatively fuses multi-turn query representations to model intent evolution, grounded in detailed video-grounded descriptions. This design effectively captures procedural nuances, surpassing existing single-turn models in accuracy and robustness. The integration of contrastive learning for cross-modal embedding alignment and the hierarchical fusion mechanism constitute key technical advancements, enabling scalable, dialogue-aware health video retrieval.
Novelty
This is the first work to explicitly incorporate multi-turn dialogue modeling into health video retrieval, leveraging detailed video-grounded descriptions and query fusion to address the limitations of single-turn systems. Unlike prior methods that focus solely on static text-video matching, this approach captures the dynamic evolution of user intent, significantly improving fine-grained semantic understanding. The combination of a two-stage retrieval process with detailed procedural descriptions represents a pioneering step in health AI, bridging the gap between natural language interaction and multimodal content understanding.
Limitations
- The system heavily relies on the quality of generated video descriptions; inaccuracies or omissions in descriptions can mislead retrieval results, especially in complex clinical scenarios.
- Multi-turn interactions are constrained by predefined query structures, limiting flexibility in handling highly ambiguous or multi-interpretative requests.
- Training and inference involve substantial computational costs, particularly for large-scale deployment, necessitating further optimization for real-time applications.
Future Work
Future research will focus on enhancing description generation quality, possibly integrating multimodal cues like audio and haptic feedback. Developing adaptive query reformulation strategies to handle more ambiguous inputs is also crucial. Additionally, efforts will aim to reduce computational overhead, enabling real-time deployment in clinical settings. Extending the framework to broader medical domains and integrating user feedback loops could further improve system robustness and personalization, pushing health AI toward more autonomous, interactive assistance.
AI Executive Summary
The rapid growth of health-related instructional videos has created a pressing need for advanced retrieval systems capable of understanding complex, evolving user queries. Traditional single-turn retrieval models, which match a static query to a set of videos, fall short in health contexts where users often require detailed, procedural, and safety-sensitive information. For example, a clinician searching for a specific rehabilitation exercise might initially query broadly but then refine their request based on posture, equipment, or injury considerations. Addressing this challenge, the paper introduces an innovative multi-turn retrieval framework that models the dynamic evolution of user intent.
Central to this approach is the construction of the Multi-Turn Health Video Retrieval Corpus (MHVRC), which pairs health instructional videos with detailed, video-grounded descriptions and subsequent refined queries. These descriptions, generated by VideoChat-Flash, emphasize visual evidence such as posture, hand placement, and equipment. DeepSeek then reformulates initial user queries into more specific follow-ups conditioned on these descriptions, simulating realistic iterative search sessions.
The core technical contribution is the Dialogue-Aware Two-Stage Retrieval (DATR) framework. The first stage employs a CLIP-style dual encoder with sparse frame sampling to perform fast, scalable coarse retrieval across large video datasets. The second stage fuses multi-turn queries using a neural fusion module and applies a lightweight cross-encoder to re-rank the top candidates, capturing procedural nuances and fine-grained semantics. This design balances efficiency with detailed understanding, enabling precise retrieval in health scenarios.
Experimental results on MHVR demonstrate that DATR outperforms strong single-turn baselines, achieving R@1 of 19.5% compared to 15.2% for HERO. The model also significantly improves the median rank and recall metrics, confirming its ability to capture procedural details. User studies further validate that multi-turn queries better express specific needs, leading to more clinically relevant results. The framework’s scalability and interpretability make it a promising foundation for future health AI applications.
This work advances health video retrieval by integrating dialogue mechanisms, detailed procedural descriptions, and multimodal fusion, addressing real-world clinical and educational needs. It opens pathways for developing intelligent, interactive health assistants capable of understanding evolving user requirements, ultimately supporting personalized medicine, remote training, and patient engagement. Despite current limitations related to description quality and computational costs, ongoing research aims to optimize and expand this approach, pushing the boundaries of health AI toward more natural, effective human-machine collaboration.
Deep Analysis
Background
Recent progress in vision-language pretraining, exemplified by models like CLIP, HERO, and CLIPBERT, has significantly improved generic text-video retrieval. However,健康视频检索面临更复杂的需求：用户不仅关注动作类别，还需细节信息如姿势、设备、受伤预防等。现有数据集如MedVidQA多为单轮问答，难以满足多轮交互的需求。健康场景中的信息需求具有动态演变特性，单轮检索难以捕获用户不断细化的意图。为此，构建专门的多轮语料库和模型成为研究重点。
Core Problem
核心问题在于如何在健康视频检索中引入多轮交互，准确理解用户在逐步细化的查询中的意图变化。传统单轮模型无法捕获细粒度的动作细节和临床上下文，导致检索结果偏离实际需求。多轮交互的挑战包括：如何有效融合多轮查询信息、如何平衡检索效率与细节匹配、以及如何构建具有健康场景特征的语料库。这些问题限制了多模态健康信息系统的实用性和智能化水平。
Innovation
本研究的创新点包括：1）构建多轮健康视频检索语料库MHVRC，结合视频grounded描述和用户后续查询，模拟真实场景；2）提出DATR双阶段模型，结合高效的双编码器与细粒度交叉编码器，有效捕获动作和细节信息；3）引入多轮查询融合机制，通过MLP和元素级操作，动态调整检索焦点。这些创新解决了单轮检索在细节捕获和意图演变中的不足，推动了健康视频检索的多轮交互研究。
Methodology
- �� 构建MHVRC：采集健康教学视频，利用VideoChat-Flash生成详细描述，强调姿势、设备、动作顺序；利用DeepSeek将初始查询转化为更具体的后续查询。
- �� Stage I：使用CLIP风格Transformer编码文本查询和视频帧，采用稀疏帧采样，生成共享嵌入空间，通过对比损失训练模型。
- �� Stage II：融合多轮查询信息，设计MLP和元素级操作，生成融合表示；对候选视频进行轻量交叉编码器评分，重排序。
- �� 训练：采用正负样本对比损失，确保语义一致性，同时优化融合机制。
- �� 评估：在MHVRC上使用Recall@K、MedR等指标，比较单轮和多轮模型性能，进行消融分析验证各组件贡献。
Experiments
使用MHVRC数据集，包含约3000个多轮查询，覆盖急救、康复、心血管等场景。基线模型包括CLIP4Clip、Frozen-in-Time、CLIPBERT和HERO。指标包括Recall@K、MedR。训练中采用批量对比损失，K值设为100，模型参数d=512。通过消融实验验证多轮融合和轻量交叉编码器的效果。用户研究评估查询改进和检索相关性，展示多轮交互的实际优势。
Results
DATR在R@1达到19.5%，优于最强单轮模型HERO的15.2%，提升4.3个百分点。R@10提升至57.1%，中位排名由37.2降至25.6。多轮查询显著增强细粒度语义表达，用户反馈显示改进的查询更具临床相关性。消融实验确认多轮融合和交叉编码器对性能的贡献，验证模型在复杂健康场景中的适用性。
Applications
该技术可应用于医疗培训、康复指导、健康教育等场景，帮助用户更精确地找到符合特定需求的教学视频。系统依赖详细描述和多轮交互，适合远程医疗、个性化康复方案制定。未来还可结合用户反馈持续优化检索效果，推动健康AI的智能化发展。
Limitations & Outlook
模型对描述质量敏感，描述遗漏或偏差会影响检索效果。多轮交互结构受限于预定义模板，难应对极端模糊或多义需求。训练和推理成本较高，模型在大规模应用中存在一定的计算压力。未来需优化描述生成和模型推理速度，以适应实际部署需求。
Plain Language Accessible to non-experts
想象你在一家厨房里准备做饭，厨房里有各种食材和工具。你想找到一份特定的菜谱，但一开始只知道大概的名字，比如‘炒菜’。你告诉助手：‘我想做辣一点的炒菜’，助手会帮你筛选出一些符合条件的菜谱，但你还可以继续补充：‘需要用到青椒和胡椒’，这样助手会根据你的新需求再次筛选。每次你提供更多细节，助手都能帮你更准确地找到最合适的菜谱。这就像论文中的多轮检索系统，先用一个大概的关键词找到一些视频，然后根据你的反馈逐步缩小范围，最终找到最符合你需求的健康教学视频。这种不断交流、逐步细化的方式，让你更快、更准地找到想要的内容。
ELI14 Explained like you're 14
想象你在学校图书馆找书，一开始只知道大概的主题，比如‘关于动物的书’，但还不确定具体想看什么。于是你跟图书馆员说：‘我想看关于猫的书’，图书馆员帮你找到一些相关的书，但你还可以说：‘我想看带有图片、适合小学生看的’，这样图书馆员会帮你筛选出更符合你需求的书。这种多轮对话就像你不断告诉助手更多细节，帮助它帮你找到最喜欢的书。论文里的系统也是这样：一开始用一个大概的关键词找到一些视频，然后你不断补充细节，比如姿势、设备、动作顺序，系统就能逐步帮你找到最合适的健康教学视频。这让搜索变得更聪明、更贴合你的实际需要。
Glossary
Dual Encoder (双编码器)
一种同时编码文本和视频的模型，将两者映射到同一空间以实现快速匹配（如CLIP模型）。
用于第一阶段的快速粗检，提升检索效率。
Cross-Encoder (交叉编码器)
一种联合编码文本和视频的模型，能捕获细粒度语义关系，适合重排序（如轻量交叉编码器）。
用于第二阶段的细粒度重排序。
Video-grounded Description (视频基础描述)
基于视频内容生成的详细动作、姿势和设备描述，用于增强语义理解。
在多轮检索中作为用户需求的语义桥梁。
Query Fusion (查询融合)
结合多轮查询信息，生成统一的语义表示以反映用户意图演变。
提升检索的细粒度匹配能力。
Recall@K (召回率@K)
检索系统在前K个结果中包含目标视频的比例，用于衡量检索覆盖度。
评估模型在健康视频检索中的效果。
Open Questions Unanswered questions from this research
- 1 如何进一步提升多轮交互中描述生成的准确性，确保描述覆盖所有关键细节，仍是未来研究的重点。
- 2 多模态信息融合（如音频、触觉）在健康视频检索中的潜力尚未充分挖掘，值得探索。
- 3 模型在极端模糊或多义需求场景下的表现仍有限，需开发更鲁棒的理解机制。
Applications
Immediate Applications
远程医疗培训
医生和患者可以通过多轮对话，快速找到符合特定康复或手术步骤的教学视频，提升培训效率。
个性化康复指导
根据患者的具体需求（如姿势、设备限制），系统能提供定制化的康复动作视频，改善治疗效果。
Long-term Vision
智能健康助理
未来可发展为全自动的健康咨询机器人，结合多模态信息，支持复杂、多轮健康问题解答，推动个性化医疗普及。
Abstract
The growing availability of health-related instructional videos creates new opportunities for clinical training, patient rehabilitation, and health education, yet existing retrieval systems remain largely single-turn: a user submits one query and receives one ranked list. This interaction is brittle in health scenarios, where information needs are often vague at first and become clinically meaningful only after follow-up constraints such as posture, hand placement, contraindications, equipment, or patient condition are specified. We introduce interactive multi-turn semantic retrieval for health videos and construct MHVRC, a Multi-Turn Health Video Retrieval Corpus, by combining video-grounded descriptions from VideoChat-Flash with query refinements generated by DeepSeek. We further propose DATR, a Dialogue-Aware Two-Stage Retrieval framework. DATR first performs efficient coarse retrieval with a CLIP-style dual encoder and sparse frame sampling, then re-ranks the top candidates through multi-turn query fusion and a lightweight cross-encoder scoring module. Experiments on MHVRC show consistent gains over strong text-video retrieval baselines, while user studies indicate that refined multi-turn queries better capture fine-grained procedural semantics than single-turn annotations. The work establishes a benchmark and a scalable technical recipe for interactive health video retrieval.