BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding
BARISTA employs dense scene graphs in egocentric videos for multi-task understanding, covering 185 real coffee-making videos with over 3.6 million annotations.
Key Findings
Methodology
This work constructs per-frame scene graphs combining object masks, bounding boxes, attributes, typed relations, and activity labels, using a semi-automated pipeline. The pipeline integrates interaction detection via CaRe-Ego, sparse manual annotation on keyframes with SAM 2, dense mask propagation with SAM 3, and identity consolidation. The scene graphs serve as a unified knowledge base for multi-task zero-shot evaluation, enabling detailed failure analysis across spatial, relational, and procedural tasks. Experiments benchmark several vision-language models (e.g., Gemini, Qwen, GPT) on object grounding, HOI recognition, relation extraction, activity classification, and temporal VQA, revealing task-specific performance gaps.
Key Results
- Qwen 3.5-27B achieved a mean average precision (mAP@50) of 0.680 on phrase grounding, outperforming other models; in HOI detection, models like Gemini 3 Flash reached F1 scores of 0.749, showing strong spatial understanding; relation extraction and activity recognition tasks exposed significant performance variation, highlighting the challenge of comprehensive scene understanding; ablation studies confirmed the importance of scene graph structure for multi-task capabilities.
- Across tasks, models demonstrated complementary strengths: spatial localization remains difficult, while relational and temporal reasoning benefit from structured annotations. The experiments underscore that no single model dominates all tasks, emphasizing the need for integrated approaches. The dense annotations notably improve zero-shot generalization, especially in complex procedural scenarios.
- The evaluation framework's flexibility allows detailed error analysis, revealing specific failure modes such as occlusion, fast motion, and ambiguous relations, guiding future model improvements.
Significance
This research advances the field of comprehensive scene understanding by providing a densely annotated egocentric dataset with a unified multi-task benchmark. It addresses the critical need for structured, interpretable evaluation of models across perception, relational reasoning, and procedural inference. The scene graph-based approach enables precise diagnosis of model weaknesses, fostering development of more robust, generalizable AI systems capable of understanding complex real-world tasks. The dataset’s focus on egocentric videos aligns with emerging applications in robotics, assistive technology, and human-computer interaction, making it a valuable resource for both academic research and industry development.
Technical Contribution
The core innovation lies in designing a semi-automated annotation pipeline that produces dense, high-quality scene graphs linking objects, relations, attributes, and activities at the frame level. This structure supports multi-task zero-shot evaluation, bridging perception and reasoning. The integration of models like SAM 2/3 for mask propagation, combined with manual verification, ensures annotation accuracy and consistency. The benchmark’s design allows systematic analysis of model capabilities and failure modes, revealing task-specific strengths and weaknesses. The approach sets a new standard for structured, multi-layered scene understanding in egocentric videos.
Novelty
BARISTA is the first to deliver dense, per-frame scene graphs for egocentric procedural videos, covering object, relation, attribute, activity, and process step annotations in a unified framework. Unlike prior datasets that focus on isolated tasks or sparse annotations, BARISTA’s comprehensive structure enables multi-task evaluation and failure diagnosis at multiple levels. Its semi-automated annotation pipeline balances scalability and quality, making dense scene graph annotation feasible at large scale. This holistic approach significantly advances the state of the art in egocentric scene understanding.
Limitations
- The annotation process relies heavily on deep models like SAM, which may introduce errors in occluded or fast-moving scenes, affecting the overall quality. The dataset is domain-specific, focusing solely on coffee preparation, limiting immediate generalization to other tasks or environments. Additionally, current models still struggle with complex relational reasoning and long-term temporal dependencies, indicating the need for more advanced temporal modeling and multi-modal fusion techniques in future work.
Future Work
Future directions include expanding scene graph annotations to other procedural domains, improving automatic annotation accuracy, and integrating multi-modal cues such as audio and gaze. Developing models with better temporal reasoning and relational understanding, possibly via transformer-based architectures, will be prioritized. Additionally, scaling the dataset and benchmark to larger, more diverse scenarios will facilitate broader applicability. Ultimately, the goal is to enable AI systems that can autonomously interpret and execute complex tasks in real-world environments, pushing towards more intelligent, adaptable agents.
AI Executive Summary
Understanding complex human activities from egocentric videos remains a significant challenge in computer vision. Traditional benchmarks often focus on isolated tasks like object detection or activity classification, lacking the structured, multi-layered understanding necessary for real-world applications. Recognizing this gap, BARISTA introduces a densely annotated egocentric dataset centered on coffee preparation—a domain rich in procedural complexity and interaction variability. The dataset comprises 185 videos with over 3.6 million annotations, including object masks, bounding boxes, attributes, typed relations, and activity labels, all linked through per-frame scene graphs. This dense annotation enables comprehensive multi-task evaluation, spanning object grounding, hand-object interaction recognition, relation extraction, activity classification, and temporal question answering, all in a zero-shot setting.
The core methodology involves a semi-automated annotation pipeline that combines deep learning models like SAM 2/3 for mask propagation with targeted manual verification, ensuring high-quality, temporally consistent scene graphs. These graphs serve as a unified knowledge base, allowing models to be evaluated across multiple tasks simultaneously and facilitating detailed failure analysis. Experiments benchmarking models such as Gemini, Qwen, and GPT reveal significant performance variations across tasks, with spatial localization and relational reasoning remaining challenging. The results highlight the importance of structured scene graphs in advancing egocentric scene understanding.
This work has broad implications for developing autonomous agents capable of understanding and executing complex procedures in real-world environments. The dense, structured annotations support both research and practical applications in robotics, assistive systems, and human-computer interaction. Despite its strengths, the approach faces limitations like domain specificity and reliance on deep models for annotation, which may introduce errors. Future efforts will focus on expanding to diverse domains, improving annotation automation, and enhancing models’ temporal and relational reasoning capabilities. Overall, BARISTA sets a new standard for structured, multi-task scene understanding in egocentric videos, paving the way for more intelligent, adaptable AI systems.
Deep Analysis
Background
场景理解一直是计算机视觉的核心问题，早期主要关注静态对象检测和分类。随着深度学习的发展，关系推理（如Scene Graph Generation）和视频理解（如EPIC-KITCHENS、Ego4D）逐步成为研究热点。现有数据集多偏重单一任务或稀疏标注，难以实现多层次的场景理解。egocentric视频因其第一人称视角，提供丰富的空间和动作信息，成为研究重点。然而，缺乏密集、结构化的标注限制了模型在复杂程序推理中的表现。VISOR、Ego4D等提供部分空间或动作信息，但未能覆盖完整关系和程序步骤。BARISTA的出现，旨在通过密集场景图标注，支持多任务评估，推动场景理解向更高层次发展。
Core Problem
当前模型在空间定位、关系推理和程序推断方面表现有限，尤其是在egocentric视频中。缺少结构化、密集的场景图，导致模型难以进行多层次推理，表现不佳。现有数据集多偏重单一任务，缺少统一评估平台，难以诊断模型缺陷。如何构建一个涵盖对象、关系、属性和程序步骤的高质量标注体系，成为亟待解决的问题。这不仅影响模型的泛化能力，也限制了其在复杂环境中的应用潜力。
Innovation
本研究创新在于提出基于场景图的密集标注体系，结合半自动化流程实现大规模高质量标注。具体包括：1）构建每帧场景图，链接对象、关系、属性和活动信息；2）利用SAM 2/3模型辅助掩码传播，确保标注时间一致性；3）引入零样本多任务评估框架，支持对象定位、关系识别、活动理解和程序推理。该体系区别于传统稀疏标注方法，为多任务、多层次场景理解提供了基础，推动了egocentric视频理解的整体提升。
Methodology
- �� 交互检测与关键帧选择：使用CaRe-Ego模型检测交互，提取交互段，选择中心帧作为标注点。• 稀疏关键帧标注：在关键帧上，人工标注对象掩码和类别，结合SAM 2提供的掩码建议。• 密集掩码传播：用SAM 2/3模型将稀疏掩码在时间上传播到中间帧，确保连续性。• 识别与身份修正：人工审核传播掩码，修正遮挡或误差，合并跨段对象身份。• 关系与活动标注：在代表帧上标注空间关系、手-物交互和程序步骤，形成完整场景图。• 任务定义：基于场景图，设计空间定位、关系识别、活动理解和时序问答任务，支持零样本评估。
Experiments
采用185个咖啡制作视频，标注总时长4.4小时，生成469K实例掩码、2.48M关系、2424个活动段。模型评估包括多种VLM（Gemini、Qwen、GPT），在对象检测、关系提取、活动识别和时序问答中进行零样本测试。指标包括mAP、F1、G-Eval等。通过不同模型规模和类型的比较，分析模型在空间定位与关系推理上的差异，验证场景图结构对提升多任务性能的作用。实验还包括消融分析，验证场景图密集标注的有效性。
Results
模型在对象定位任务中，Qwen 3.5-27B达到mAP@50 0.680，优于其他模型。关系提取任务中，模型表现差异明显，表明关系理解仍是瓶颈。空间定位任务对模型空间感知要求高，关系推理依赖丰富的场景图信息。多任务评估揭示模型在不同任务中的能力互补，未出现单一优势模型。消融实验显示，场景图密集标注显著改善模型推理能力，尤其在零样本条件下效果更明显。
Applications
该数据集和评估框架可应用于机器人自主操作、智能监控、虚拟助手等场景，提升模型在复杂环境中的空间感知和程序推理能力。未来，结合多模态信息和强化学习，有望实现更智能的自主系统，推动人机交互和自动化水平提升。
Limitations & Outlook
当前标注依赖深度模型辅助，可能在遮挡或快速运动场景中出现误差；仅覆盖咖啡制作场景，泛化到其他复杂任务仍需扩展；模型在关系推理和程序推断方面仍存在不足，未来需引入更强的时序建模和多模态融合技术。
Plain Language Accessible to non-experts
想象你在厨房准备咖啡。每个动作，比如拿咖啡豆、放入研磨机、按下按钮，都是一段复杂的舞蹈。科学家们用一种超级相机，把每个动作、每个物体的位置、它们之间的关系都详细画出来，就像画一张流程图。这张图不仅告诉你谁在做什么，还能显示动作之间的联系，比如先倒咖啡粉，再压紧，然后放入机器。通过这样的详细地图，电脑可以学会理解整个咖啡制作的流程，甚至自己判断下一步该做什么。这就像教会机器人看懂厨房的每个细节，让它变得更聪明、更会自己做事。
ELI14 Explained like you're 14
嘿，你知道吗？想象你在厨房里做咖啡。你会用手拿咖啡豆、放到研磨机里，然后按按钮，最后倒出香喷喷的咖啡。现在，科学家们也在试图让电脑学会像你一样理解这个过程。他们用一种特别的方法，把每个动作、每个物体的位置都画成一张超级详细的地图，就像你画的流程图一样。这张地图告诉电脑：谁在做什么，谁和谁有关系，动作是怎么连接的。这样，电脑就能学会看懂整个咖啡制作的步骤，甚至帮你提醒下一步该做什么。就像让机器人变得更聪明，能自己理解厨房里的事情一样。是不是很酷？
Abstract
Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a scene. Yet understanding physical processes remains difficult, as models must combine object localization, hand-object interactions, relational parsing, temporal reasoning, and step-level procedural inference. Existing benchmarks usually evaluate these capabilities separately, limiting diagnosis of why models fail on procedural tasks. We introduce BARISTA, a densely annotated egocentric dataset and benchmark of 185 real-world coffee-preparation videos covering fully automatic, portafilter-based, and capsule-based workflows. BARISTA provides verified per-frame scene graphs linking persistent object identities to masks, tracks, boxes, attributes, typed relations, hand-object interactions, activities, and process steps. From these graphs, we derive zero-shot language-based tasks spanning phrase grounding, hand-object interaction recognition, referring, activity recognition, relation extraction, and temporal visual question answering. Experiments reveal strong variation across task families and no consistently dominant model family, positioning BARISTA as a challenging diagnostic benchmark for procedural video understanding. Code and dataset available at https://huggingface.co/datasets/ramblr/BARISTA.