From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models
Proposes TAG, converting egocentric videos into structured temporal graphs for zero-shot action recognition using VLMs, outperforming traditional pixel-based methods.
Key Findings
Methodology
The approach employs multi-stage prompting to transform short video segments into natural language descriptions, which are then parsed into structured open-vocabulary relation triplets. These triplets are assembled into a temporal graph serialized as text. The VLM then reasons over this symbolic sequence for action classification, without any training. Experiments on EGTEA and Epic-Kitchens-100 datasets involved 11 models from 2B to 235B parameters, demonstrating that structured graphs enable efficient in-context learning and outperform frame-based inference, especially in few-shot settings.
Key Results
- On EGTEA, graph-based reasoning improved Top-1 accuracy by approximately 9% across models, with larger gains (over 10%) in models like Qwen-3.5B. On Epic-Kitchens-100, combining graphs with in-context examples led to over 15% accuracy increase, notably in action recognition. The results held across model sizes, showing robustness and scalability.
- Across models from 2B to 235B parameters, structured reasoning demonstrated better generalization and robustness, even when pretraining data favored pixel memorization. Few-shot demonstrations significantly boosted performance, confirming the high information compression and rapid adaptation capacity of the symbolic graphs.
- Ablation studies indicated that window size, description detail, and relation attributes critically impact accuracy. The approach consistently outperformed raw frame inputs, especially with minimal training or fine-tuning, highlighting the method's efficiency.
Significance
This work shifts the paradigm from perception-centric to reasoning-centric video understanding, leveraging symbolic representations to improve interpretability and generalization. It addresses fundamental limitations of current VLMs in dynamic, fine-grained action recognition, paving the way for applications in robotics, AR, and surveillance that require robust, data-efficient reasoning. The method's training-free nature makes it scalable and adaptable across diverse scenarios, reducing reliance on large annotated datasets and enabling better utilization of pre-trained models’ latent reasoning capabilities.
Technical Contribution
The paper introduces a novel multi-stage prompting pipeline that converts videos into structured temporal graphs without training. It combines natural language narrative generation, symbolic parsing into relation triplets, and sequence serialization for language-based reasoning. This approach decouples perception from reasoning, exploiting the pre-trained VLMs’ strengths in symbolic inference. The framework supports few-shot and zero-shot action recognition, demonstrating significant improvements over traditional pixel-based methods and establishing a scalable, training-free paradigm for dynamic scene understanding.
Novelty
This is the first comprehensive framework to convert egocentric videos into structured, open-vocabulary temporal graphs for reasoning with pre-trained VLMs. Unlike prior work relying on supervised graph annotations or end-to-end training, this method leverages multi-stage prompting to generate symbolic representations, enabling efficient in-context learning. It effectively decouples perception from reasoning, addressing the limitations of static image pretraining and pushing forward the frontier of symbolic video understanding.
Limitations
- The method's accuracy depends heavily on the quality of natural language descriptions, which can be affected by visual ambiguities or occlusions, leading to potential errors in relation extraction.
- Handling complex multi-hand interactions and cluttered backgrounds remains challenging, as the current description parser may oversimplify or miss subtle relations.
- Constructing and reasoning over long-duration videos may incur high computational costs, requiring further optimization for real-time applications.
Future Work
Future directions include integrating dynamic graph learning to adaptively refine relation representations, incorporating multimodal cues like audio for richer descriptions, and developing real-time processing capabilities. Exploring end-to-end training with weak supervision or semi-supervised signals could further enhance accuracy and robustness, enabling deployment in more complex, real-world scenarios.
AI Executive Summary
Understanding human actions in egocentric videos is a complex challenge due to the fine-grained hand-object interactions and subtle temporal transitions involved. Traditional models, primarily based on end-to-end visual feature extraction, often struggle to accurately capture these dynamics, especially in zero-shot or few-shot scenarios. This paper introduces a novel framework called Temporal Action Graph (TAG), which fundamentally rethinks video understanding by converting raw video frames into structured, symbolic temporal graphs. The process begins with dividing videos into overlapping short segments, each processed by a vision-language model (VLM) to generate natural language descriptions of the dominant interactions. These descriptions are then parsed into relation triplets—source, relation, object—and assembled into a comprehensive, serialized temporal graph. This graph effectively encodes the evolution of hand-object interactions over time, serving as an interpretable, open-vocabulary representation.
The key innovation lies in leveraging the pre-trained VLMs’ strong reasoning capabilities within the symbolic domain, bypassing the need for costly fine-tuning. The approach employs multi-stage prompting, where the model first generates semantic narratives, then formalizes them into structured graphs. This decoupling of perception and reasoning allows the models to perform highly efficient in-context learning by simply prepending a few labeled graph examples, dramatically boosting accuracy in few-shot settings. Extensive experiments on EGTEA and Epic-Kitchens-100 datasets across 11 models ranging from 2 billion to 235 billion parameters demonstrate that graph-based reasoning consistently outperforms traditional frame-based inference, with improvements exceeding 10% in many cases.
The results highlight the potential of symbolic reasoning in dynamic scene understanding, especially for applications like robotics, augmented reality, and intelligent surveillance. The approach’s training-free nature, scalability, and robustness to pretraining biases mark a significant step forward. Despite current limitations in handling complex multi-hand interactions and long videos efficiently, the framework opens promising avenues for future research—such as adaptive graph learning, multimodal integration, and real-time deployment—aiming to make machines better at understanding the nuanced flow of human actions in everyday life.
Deep Analysis
Background
视频理解技术在过去十年经历了快速发展，从早期的手工特征到深度学习模型如C3D、I3D、SlowFast等。近年来，符号化表示如场景图和关系图逐渐应用于静态和动态场景，提升了关系推理能力。视觉-语言模型（如CLIP和ALIGN）在静态图像理解中取得突破，但在视频动态场景中的应用仍受限，主要因其训练偏向静态场景。多模态融合和符号推理成为研究热点，旨在解决模型对时间动态理解不足的问题。egocentric视频因其微妙的手部动作和交互关系，成为研究的难点，亟需更细粒度的动态关系建模。
Core Problem
现有VLM在细粒度动作识别中表现有限，主要问题包括：1)模型难以捕获手-物交互的微妙变化，2)对动作的时间演变理解不足，3)端到端训练成本高，泛化能力有限。在egocentric场景中，手部动作和物体关系的微小变化决定了动作类别，现有方法难以准确捕捉这些信息，限制了其实际应用。如何在无需大量标注的情况下，提升模型对复杂动态关系的理解，成为核心难题。
Innovation
本研究的创新点包括：1)多阶段提示策略，将视频划分为短时段，生成描述性文本，增强模型的符号表达能力；2)解析描述为关系三元组，形成结构化关系图；3)将关系图序列化为文本，利用预训练VLM进行推理，避免微调；4)引入少样本示例增强推理效果。这些创新实现了视频到符号空间的无缝转换，有效解耦感知与推理，显著提升了模型在零-shot和少-shot场景中的表现。
Methodology
- �� 视频采样：将连续视频划分为重叠的时间窗口（每个窗口4帧），确保捕获细粒度动作变化。
- �� 描述生成：用VLM对每个窗口生成自然语言描述，突出手部动作和交互对象。
- �� 关系解析：将描述解析为关系三元组（源、关系、对象），属性包括手部角色、动作类型和对象属性。
- �� 时间图拼接：将局部关系图按时间顺序拼接，形成完整的时间序列关系图。
- �� 序列化推理：将关系图序列化为文本输入，利用VLM进行动作类别预测。
- �� 少样本示例：在提示中加入少量标注的关系图示例，提升模型推理准确性。
Experiments
在EGTEA和Epic-Kitchens-100两个公开数据集上，评估11个不同参数规模的VLM（从2B到235B参数）。比较纯视觉输入、关系图输入和结合少样本示例的多模态输入效果。指标包括Top-1、Top-3、Top-5准确率，特别关注少样本学习能力。还进行了不同时间窗口大小、关系属性丰富度的消融分析，验证方法的鲁棒性和扩展性。实验结果显示，结构化关系图在所有模型中均优于纯帧输入，表现出更强的泛化能力和鲁棒性。
Results
结构化图推理在所有模型中均优于纯视觉输入，提升幅度超过10%。大模型（如Qwen-3.5B）在图+ICL条件下，Top-1准确率达54%以上，显著优于传统方法。少样本示例的引入极大提升了模型在少样本场景中的表现，验证了符号化表示的高效信息压缩和快速适应能力。多参数模型在复杂动作识别中表现出更强的鲁棒性，表明符号推理在动态场景中的潜力。
Applications
该方法适用于机器人交互、增强现实、智能监控等场景，尤其在缺乏大量标注数据的环境中表现出优势。通过符号关系图，系统可以更好理解复杂动作和关系，提升场景理解和决策能力。未来结合实时处理和动态图结构，有望实现更高效的动态场景分析，为智能系统赋能。
Limitations & Outlook
当前方法在遮挡、多手交互和复杂背景下仍有局限，描述生成受视觉模糊影响，关系解析可能出错。符号化过程依赖预训练模型的描述能力，偏差可能引入错误。长视频关系图构建和推理存在计算瓶颈，未来需优化算法以提升效率。
Plain Language Accessible to non-experts
想象你在厨房里做饭，视频就像你用手机拍下的整个过程。传统的方法就像是用眼睛看一眼，然后试图记住所有细节，但很容易遗漏关键步骤。现在，这个新方法像是请一个聪明的助手，把你每个动作用简单的句子写下来，比如“拿起刀”、“切菜”，然后把这些句子整理成一份时间表，告诉你每个动作发生的顺序。这样，即使你没有看视频，也能通过这份清单快速知道你做了什么。它把复杂的视觉信息变成简单的文字和关系图，就像你用笔记整理厨房操作一样，让电脑更容易理解动作的流程。这不仅让电脑变得更聪明，还能帮它更快学会新动作，就像你用笔记记厨房步骤一样，特别方便。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每个动作都很细微，比如拿东西、放东西、转身。以前的电脑就像用眼睛看一遍，然后试图记住所有细节，但很难抓住那些微妙的变化。现在，这个新方法就像是你用一支神奇的笔，把每个动作用简单的句子写下来，比如“你用左手拿起杯子”，然后把这些句子按时间顺序整理成一份清单。电脑可以看这份清单，知道你做了什么，比单纯看图片更聪明。它还能用这份清单猜出你下一步会做什么，就像你用笔记猜下一步动作一样。这让电脑变得更聪明，能更好理解复杂的动作，就像你用笔记帮你记住游戏中的每个动作一样。是不是很酷？
Glossary
Temporal Action Graph (时间动作图)
一种将视频中动作关系和时间演变结构化的符号表示，用关系三元组描述手-物交互，便于推理。
用来将视频转化为符号化的时间序列关系结构。
Vision-Language Model (视觉-语言模型)
预训练模型，能同时理解图像和文本，支持跨模态推理。
用作视频动作推理的基础工具。
In-Context Learning (上下文学习)
通过在提示中加入示例，让模型在无需微调的情况下学习新任务。
用少样本示例增强模型推理能力。
关系三元组 (Relation Triplet)
由源、关系、对象组成的结构化关系，用于描述场景中的交互。
构建时间图的基本单元。
多阶段提示 (Multi-stage Prompting)
分步骤引导模型生成描述和关系，提升符号化的准确性。
实现视频到关系图的自动转换。
Open Questions Unanswered questions from this research
- 1 如何进一步提升复杂场景中多手交互的关系解析准确率，仍需研究更强的描述生成和关系抽取机制。
- 2 符号化推理在长时序视频中的效率与效果，尚未充分验证，未来需优化图的压缩和推理速度。
- 3 模型偏差和错误传播问题，尤其在未见类别和模糊场景中，仍是挑战。
Applications
Immediate Applications
智能监控
利用符号关系图快速识别异常动作，提升安全监控效率，降低误报率。
机器人交互
帮助机器人理解人类微妙手势和动作，增强人机协作能力。
Long-term Vision
自动场景理解
实现全自动化的动态场景分析，应用于自动驾驶、虚拟现实等领域，推动智能环境的普及。
Abstract
Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions, a task where general-purpose Vision-Language Models (VLMs) often struggle when operating directly on raw pixels. We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs. In a multi-stage prompting pipeline, we first generate dense natural language narratives over short temporal windows as a semantic bottleneck, then formalize them into structured, open-vocabulary graph representations. On the EGTEA and Epic-Kitchens-100 datasets, the symbolic representation unlocks efficient in-context learning: few-shot graph demonstrations yield substantial accuracy gains over zero-shot frame and graph-based inference alike. Even in the zero-shot setting, graph-based reasoning remains competitive with pixel-based inference despite potential pretraining contamination favoring the latter. Across 11 open-weight VLMs from 6 model families ranging from 2B to 235B parameters, our findings indicate that current VLMs are more effective as symbolic reasoners than as direct visual observers. By projecting video into the language domain, we provide a scalable, fine-tuning-free alternative to end-to-end approaches that better leverages these models' latent reasoning strengths. The code will be made public.