StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation
StillFast introduces an end-to-end model for short-term object interaction anticipation, achieving 13.29% Top-5 mAP on EGO4D v2, surpassing SOTA.
Key Findings
Methodology
This paper proposes StillFast, a dual-branch deep network combining a high-resolution still image branch (ResNet-50) and a low-resolution, high-frame-rate video branch (X3D-M). Features from both branches are fused via a feature pyramid to capture multi-scale spatial and temporal cues. The model employs a Region Proposal Network (RPN) and RoI align for local feature extraction, integrating global scene context through average pooling. It predicts future object bounding boxes, classes, interaction verbs, and time-to-contact (TTC) in a single end-to-end training process, optimizing multiple loss functions simultaneously. This design simplifies training, enhances prediction accuracy, and improves efficiency over traditional two-stage methods.
Key Results
- On the large-scale EGO4D v2 dataset, StillFast achieved a Top-5 mAP of 13.29% for Noun+Verb+TTC, outperforming the baseline FRCNN+SF (9.45%) by 3.84%. In v1, it also outperformed baselines with a significant margin, demonstrating strong generalization. Ablation studies confirmed the effectiveness of feature fusion and end-to-end training, with notable improvements in verb and TTC predictions.
- The model's single-stage training reduced computational cost and training time, making it scalable for larger datasets. Results indicate that the unified architecture effectively captures both spatial details and motion cues, leading to superior performance across multiple metrics.
- Experimental results validate the approach's robustness, with consistent improvements across different versions of the dataset, confirming its potential for real-world deployment in autonomous systems and smart environments.
Significance
This work advances the state of the art in short-term object interaction anticipation, addressing the limitations of multi-stage pipelines. By enabling accurate, real-time predictions of future interactions, it opens new avenues for intelligent automation in autonomous driving, robotics, and industrial safety. The unified end-to-end framework simplifies deployment and fosters further research into multi-modal, multi-task video understanding, contributing significantly to the development of proactive AI systems.
Technical Contribution
The key innovation lies in the integration of dual spatial-temporal feature extraction with a unified feature pyramid, enabling simultaneous detection and prediction tasks. The model's end-to-end training paradigm reduces complexity, improves feature sharing, and enhances prediction accuracy. It introduces a novel fusion strategy that combines high-res static and low-res dynamic information, setting a new standard for efficiency and performance in short-term interaction forecasting.
Novelty
This is the first application of a combined still-image and video processing architecture for short-term object interaction anticipation. Unlike previous two-stage approaches, StillFast unifies detection and prediction in a single network, leveraging multi-scale feature fusion and end-to-end optimization. Its innovative design addresses the core challenge of jointly modeling spatial and temporal cues for proactive behavior prediction.
Limitations
- The model's robustness under extreme occlusion or rapid scene changes remains limited, potentially affecting prediction accuracy. Its performance depends heavily on large annotated datasets, which may limit applicability in less-studied environments.
- Multi-task training introduces optimization challenges, sometimes leading to unstable convergence or performance trade-offs among tasks. Real-time deployment requires further model compression and acceleration.
- Future work should explore multi-modal data integration, model lightweighting, and adaptation to diverse real-world scenarios to enhance robustness and scalability.
Future Work
Future directions include incorporating multimodal inputs such as audio and sensor data, developing lightweight architectures for real-time deployment, and extending the framework to long-term interaction prediction. Additionally, exploring self-supervised learning and domain adaptation techniques could further improve generalization across diverse environments.
AI Executive Summary
Predicting human-object interactions in the near future is a fundamental challenge in autonomous perception systems. Existing approaches often rely on multi-stage pipelines, involving separate detection and action prediction modules, which can be inefficient and less accurate. To address these issues, this paper introduces StillFast, an innovative end-to-end neural architecture designed for short-term object interaction anticipation from egocentric videos.
StillFast employs a dual-branch backbone inspired by SlowFast networks, combining a high-resolution still image pathway with a low-resolution, high-frame-rate video pathway. These features are fused via a multi-scale feature pyramid, enabling the model to effectively capture both spatial details and motion cues. The architecture integrates a Region Proposal Network (RPN) and RoI align for local feature extraction, while global scene context is incorporated through average pooling. The model jointly predicts the location, category, interaction verb, and time-to-contact (TTC) of future active objects, all within a single training process.
Experimental results on the large-scale EGO4D dataset demonstrate the effectiveness of StillFast. It achieved a Top-5 mAP of 13.29% on the combined task of noun, verb, and TTC prediction, outperforming previous state-of-the-art methods such as FRCNN+SF by a significant margin. The ablation studies confirmed that feature fusion and end-to-end training are critical for performance gains. The approach also offers practical advantages, including reduced training complexity and improved scalability.
This research marks a significant step forward in proactive scene understanding, with broad implications for autonomous driving, industrial safety, and assistive robotics. By accurately forecasting future interactions, systems can respond more intelligently and safely. Limitations include robustness under occlusion and scene variability, which are areas for future exploration. Overall, StillFast provides a powerful, unified framework for real-time, high-precision short-term behavior prediction, paving the way for more intelligent autonomous systems.
Deep Dive
Abstract
Anticipation problem has been studied considering different aspects such as predicting humans' locations, predicting hands and objects trajectories, and forecasting actions and human-object interactions. In this paper, we studied the short-term object interaction anticipation problem from the egocentric point of view, proposing a new end-to-end architecture named StillFast. Our approach simultaneously processes a still image and a video detecting and localizing next-active objects, predicting the verb which describes the future interaction and determining when the interaction will start. Experiments on the large-scale egocentric dataset EGO4D show that our method outperformed state-of-the-art approaches on the considered task. Our method is ranked first in the public leaderboard of the EGO4D short term object interaction anticipation challenge 2022. Please see the project web page for code and additional details: https://iplab.dmi.unict.it/stillfast/.