OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs
Introduces OmniAssistBench, a dataset built via reverse engineering of internet videos, to evaluate Omni-LLMs in multi-turn multimodal interactions.
Key Findings
Methodology
This work constructs the dataset by reverse-engineering online videos, extracting logical user goals and fixed interaction paths. It involves expert annotation of QA pairs, video editing to embed prompts, and a two-tier task framework covering basic perception and advanced goal-oriented tasks. The approach ensures path consistency, avoiding divergence in interaction trajectories. Evaluation uses models like Gemini-3-Pro and Qwen3-Omni-Instruct, with metrics assessing visual prompt understanding, long-term memory, and delayed responses. The methodology emphasizes manual annotation for subtle details and simulates continuous multi-turn interactions, overcoming static dataset limitations.
Key Results
- Gemini-3-Pro scored 66.4 out of 100, while Qwen3-Omni-Instruct scored 51.2, indicating significant room for improvement in multimodal understanding and interaction continuity.
- Models struggled with visual prompts such as hand gestures, maintaining context over multiple turns, and delaying responses until appropriate visual cues appeared.
- The evaluation revealed that current models often forget initial user goals in extended interactions and respond prematurely or incompletely, highlighting key challenges for real-world deployment.
Significance
This benchmark addresses a critical gap in evaluating multimodal large models in realistic, multi-turn scenarios. By leveraging reverse-engineered videos and fixed interaction paths, it provides a standardized, rigorous framework to measure and compare model capabilities. The insights gained guide future research towards models that can better understand visual cues, maintain context, and respond appropriately over time, accelerating progress toward reliable AI assistants in complex environments.
Technical Contribution
The paper introduces a novel data construction pipeline based on reverse engineering, ensuring fixed interaction paths through prior knowledge. It designs a two-tier task framework and develops a comprehensive scoring system that evaluates perception, memory, and response delay. This approach enables systematic assessment of models' multimodal understanding in realistic scenarios, providing a new benchmark for the community. The methodology bridges the gap between static datasets and dynamic interaction requirements, fostering innovation in multimodal AI.
Novelty
First to systematically utilize reverse-engineered internet videos for multi-turn interaction dataset construction, with explicit path priors to guarantee path fidelity. This approach overcomes the limitations of traditional static datasets, enabling evaluation of models in more realistic, diverging interaction scenarios. The dual-tier task design and manual annotation process represent a significant advancement in benchmarking multimodal interactive AI.
Limitations
- Data construction relies heavily on manual expert annotation, making it labor-intensive and difficult to scale rapidly.
- Models still face challenges in visual prompt comprehension and long-term memory retention, especially in complex multi-turn interactions.
- The dataset covers a limited set of fixed paths, which may not fully represent the diversity of real-world interactions. Future work should incorporate multiple paths and automate parts of the data generation process.
Future Work
Future efforts will focus on automating data annotation, expanding interaction paths, and improving model architectures for better visual understanding and memory. Integrating multimodal reasoning and reasoning over longer contexts will further enhance model robustness. The community can adopt this benchmark to develop models capable of more natural, flexible, and reliable human-AI interactions in real-world scenarios.
AI Executive Summary
The rapid development of multimodal large language models (Omni-LLMs) has opened new horizons for real-time video-based assistance. However, evaluating these models in realistic, multi-turn interactive scenarios remains a major challenge. Traditional datasets are static, capturing only isolated responses, and cannot simulate the dynamic influence of model responses on user actions. To address this, the authors introduce OmniAssistBench, a novel benchmark constructed through reverse engineering of internet videos.
This approach involves meticulously analyzing publicly available videos to deduce logical user goals and fixed interaction paths. Experts manually annotate question-answer pairs, segment videos into multi-turn clips, and embed prompts such as speech and gestures. This pipeline ensures the interaction trajectory remains consistent, enabling precise evaluation of models' understanding and response capabilities. The benchmark features a two-tier framework: the basic tier assesses fundamental perception skills, while the advanced tier evaluates complex, goal-oriented tasks in realistic scenarios.
Experimental results demonstrate that state-of-the-art models like Gemini-3-Pro achieve scores around 66, while open-source models score below 52. These findings reveal significant gaps, especially in visual prompt comprehension, long-term memory, and delayed responses. The evaluation uncovers that current models often forget initial goals over multiple turns, respond prematurely, or misinterpret visual cues, highlighting critical areas for improvement.
This work provides a vital step toward developing reliable, human-like assistants capable of understanding nuanced multimodal interactions. By establishing a rigorous, manually curated dataset and evaluation framework, it sets a new standard for future research. The authors suggest that automating data generation and expanding interaction diversity will further enhance model robustness, ultimately enabling AI systems that can seamlessly assist in complex real-world tasks.
Deep Analysis
Background
Recent advances in multimodal large language models (e.g., GPT-4, PaLM-E) have demonstrated impressive capabilities in static content understanding. However, real-world applications demand models that can engage in continuous, multi-turn interactions involving visual, auditory, and textual cues. Existing datasets like VQA and VideoQA focus on single-turn questions, lacking the temporal and contextual depth needed for interactive scenarios. Traditional evaluation methods rely on offline static datasets, which cannot simulate the dynamic influence of model responses on user behavior. This gap hampers progress toward practical AI assistants capable of real-time, multi-modal understanding and response. Constructing datasets that accurately reflect real interaction complexity is thus a pressing need, but current approaches are limited by manual effort and lack of path fidelity, motivating the development of new methodologies.
Core Problem
The core challenge lies in evaluating models' abilities to understand and respond in multi-turn, multimodal interactions that mirror real-world scenarios. Static datasets cannot capture the causal influence of model responses on subsequent user actions, leading to unreliable assessments. Additionally, models often struggle with visual prompt comprehension, maintaining context over multiple turns, and delaying responses until the appropriate visual cues appear. These issues are compounded by the diversity of possible interaction paths, making it difficult to design comprehensive benchmarks. Without a standardized, realistic evaluation framework, progress in developing reliable, human-like assistants remains limited. The problem is further exacerbated by the labor-intensive nature of manual data annotation, which constrains dataset scale and diversity.
Innovation
This work introduces a pioneering approach to dataset construction by reverse engineering internet videos, extracting logical user goals and fixed interaction paths. It ensures path fidelity through prior knowledge, preventing divergence in interaction trajectories. The two-tier task framework separates fundamental perception skills from complex goal-oriented tasks, enabling targeted evaluation. Manual expert annotation guarantees subtle detail accuracy, while video editing and speech embedding simulate realistic multi-turn interactions. The scoring system combines semantic similarity and key point matching, providing a comprehensive performance measure. This methodology bridges the gap between static datasets and dynamic, real-world interactions, offering a scalable yet precise evaluation platform that can adapt to diverse scenarios.
Methodology
- �� 视频筛选：从YouTube等平台筛选符合特定剧情的交互视频。• 目标推导：分析视频内容，推断用户目标和路径先验。• QA设计：由专家手工设计问答对，确保路径唯一性。• 视频编辑：嵌入语音提问、视觉提示（手势、书写）模拟连续交互。• 数据标注：标注关键点和语义标签，确保多轮交互的连贯性。• 评分体系：结合语义相似度和关键点匹配，客观评价模型表现。• 模型评估：采用Gemini-3-Pro和Qwen3-Omni-Instruct，分析模型在视觉提示理解、上下文保持和延迟反应的能力。
Experiments
采用300个多轮交互视频，涵盖7个主要任务和16个子任务，专家标注问答对。模型在不同任务中表现差异明显，Gemini-3-Pro得分66.4，Qwen3-Omni-Instruct得分51.2。通过消融实验验证路径先验的有效性，分析模型在视觉提示（如手势）和长时记忆中的不足。多轮复杂场景中，模型易遗忘用户目标，响应不及时，验证了评估体系的有效性和模型的改进空间。
Results
模型在视觉提示理解方面表现不足，尤其在手势识别上准确率低于70%。长时记忆保持能力有限，模型在多轮交互中遗忘早期目标。延迟响应方面，模型常提前或过早回答，影响交互自然性。整体评分显示，当前模型仍难以满足复杂场景的需求，未来需在多模态融合和记忆机制上突破。
Applications
该评估平台可用于开发更智能、更可靠的实时视频助手，广泛应用于智能家居、辅助驾驶、远程教育等场景。模型需具备多模态理解、连续记忆和延迟响应能力，才能满足实际需求。未来，结合自动化数据生成，将推动模型在复杂、多变环境中的应用落地。
Limitations & Outlook
数据构建依赖专家手工，耗时长，难以快速扩展。模型在视觉提示和长时记忆方面仍表现不足，难以应对多样化场景。未来需引入自动化标注和多路径训练，提升模型鲁棒性和适应性。
Plain Language Accessible to non-experts
想象你在一家厨房里做饭。每次你需要准备食材、按照步骤操作，还要记住之前做过的事情。传统方法就像只看菜谱，告诉你每一步，但不能根据你手边的食材变化调整。现在的智能助手就像一个聪明的厨师，不仅知道菜谱，还能根据你厨房里的情况实时指导你。这个研究就是在教电脑怎么像厨师一样，边看视频边帮你做饭，确保每一步都正确，记住你之前做过的事情，还能等你准备好再回答问题。它用一种特殊的方法，把很多厨房视频拆开，模拟你和厨师的对话，帮你学会做各种菜肴。
ELI14 Explained like you're 14
想象你在学校参加一个科学实验，你需要跟老师和同学们一起完成一系列步骤。每次你问问题，老师会根据你的问题和之前的实验情况给你答案，但如果你没有记住之前的内容，可能会搞错。这个研究就像在教电脑如何在实验中帮你记住每个步骤，理解你说的话，还能等你准备好再回答。它用一种特别的方法，把很多实验视频拆成一段段，让电脑学会像人一样跟你互动。虽然现在还不完美，但未来它会变得更聪明，能帮你做更复杂的事情，就像一个真正的助手一样。
Abstract
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.