PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning
PaSBench-Video is a 740-video benchmark for testing proactive safety warning capabilities of video MLLMs; no model exceeds 20.0% on strictest metric.
Key Findings
Methodology
The study uses a benchmark dataset, PaSBench-Video, containing 740 videos across driving, healthcare, daily life, and industrial production. Risk videos are annotated with frame-level risk onset and accident boundaries. Models must causally observe streaming video and issue temporally calibrated, content-correct warnings.
Key Results
- No model exceeds 20.0% on the strictest metric, with recall tightly coupled with false-positive rate (Pearson correlation 0.64).
- Models achieve moderate recall at low false-positive rates in daily life but perform poorly in driving.
- Models rely on scene-level activity cues rather than reasoning about emerging harm.
Significance
This study introduces the PaSBench-Video benchmark, addressing the gap in evaluating video MLLMs' proactive safety warning capabilities. It emphasizes the importance of temporal calibration and content accuracy, revealing performance disparities across domains.
Technical Contribution
PaSBench-Video provides a structured testbed allowing independent evaluation of models' temporal risk reasoning capabilities. It incorporates streaming evaluation, temporal calibration, and false-positive robustness.
Novelty
This is the first benchmark focusing on proactive safety warning capabilities of video MLLMs, particularly in streaming video environments.
Limitations
- Current models perform poorly in driving, with high false-positive rates.
- Models primarily rely on scene-level activity cues.
- Not yet deployment-ready due to engineering and capability constraints.
Future Work
Future work could focus on improving models' temporal reasoning capabilities, reducing false-positive rates, and exploring more efficient engineering implementations.
AI Executive Summary
In many safety-critical accidents, visually recognizable warning signs often precede the incident. Existing video multimodal large language models (MLLMs) can serve as proactive safety assistants, issuing warnings before accidents occur. However, current benchmarks fail to adequately evaluate this capability. The PaSBench-Video benchmark fills this gap by providing a dataset of 740 videos across driving, healthcare, daily life, and industrial production. The study finds that models perform poorly on the strictest metric, with recall tightly coupled with false-positive rate, and exhibit significant performance disparities across domains. In particular, models struggle to distinguish routine from hazardous scenes in driving. This indicates that current models rely on scene-level activity cues rather than reasoning about emerging harm. Future research directions include improving models' temporal reasoning capabilities and reducing false-positive rates.
Deep Analysis
Background
With the development of multimodal large language models, video analysis holds great potential for safety monitoring. However, existing benchmarks focus mainly on static input evaluation, lacking tests for proactive safety warning capabilities in streaming video. The PaSBench-Video benchmark fills this gap by providing a structured dataset.
Core Problem
Current multimodal large language models perform poorly in identifying and warning of potential risks, especially in streaming video environments. Models need to accurately identify risks and issue warnings without compromising user trust.
Innovation
PaSBench-Video's innovation lies in its structured design, allowing independent evaluation of models' temporal risk reasoning capabilities. It incorporates streaming evaluation, temporal calibration, and false-positive robustness.
Methodology
- �� Use PaSBench-Video dataset for evaluation
- �� Models must causally observe video and issue warnings
- �� Evaluation metrics include temporal calibration and content accuracy
- �� Test 13 multimodal large language models
Experiments
Experiments use the PaSBench-Video dataset, covering videos from four domains. Models must causally observe streaming video and issue warnings. Evaluation metrics include recall, false-positive rate, and temporal calibration.
Results
The study finds that models perform poorly on the strictest metric, with recall tightly coupled with false-positive rate. In particular, models struggle to distinguish routine from hazardous scenes in driving.
Applications
PaSBench-Video can be used to evaluate the application potential of video MLLMs in safety monitoring, especially in scenarios requiring proactive safety warnings.
Limitations & Outlook
Current models perform poorly in driving, with high false-positive rates. Models primarily rely on scene-level activity cues rather than reasoning about emerging harm.
Plain Language Accessible to non-experts
Imagine you have a smart camera installed at home that monitors everything. When a baby starts climbing the crib rail, the camera will issue a warning to alert you. It's like having a vigilant babysitter who helps you take action before an accident occurs. The PaSBench-Video benchmark is designed to test the alertness of these smart cameras, checking if they can accurately identify risks and issue warnings in different scenarios.
ELI14 Explained like you're 14
Imagine you're playing a game with a robot assistant that warns you when you're in danger. This robot needs to be super smart, able to alert you before you even notice the danger. The PaSBench-Video benchmark is like a testing ground to see if these robot assistants can accurately identify dangers and issue warnings in different game scenarios.
Glossary
Multimodal Large Language Models
Models that combine multiple input modes (e.g., text, images, video) for complex tasks.
Used for video analysis and proactive safety warning.
Streaming Video
Continuously playing video stream allowing real-time analysis.
Used to test models' real-time risk detection capabilities.
False Positive Rate
Frequency at which a model incorrectly issues warnings.
Used to evaluate model accuracy and reliability.
Temporal Calibration
Precision of a model's warning timing.
Used to assess models' temporal sensitivity.
Risk Detection
Ability to identify potential dangers and issue warnings.
A core function of the models.
Open Questions Unanswered questions from this research
- 1 How to improve models' risk detection capabilities in driving? Current methods struggle to distinguish routine from hazardous scenes.
- 2 How to reduce false-positive rates? Frequent false alarms undermine user trust.
Applications
Immediate Applications
Home Monitoring
Smart cameras can be used to monitor home safety and issue timely warnings.
Long-term Vision
Autonomous Driving Safety
Enhancing the safety of autonomous driving systems to reduce accidents.
Abstract
Between the first visible sign of danger and the moment an accident occurs, there is often a window where intervention remains possible. Video-capable multimodal large language models (MLLMs) could serve as always-on safety monitors that issue warnings during this window. Yet current benchmarks do not test this ability: they rely on static inputs, ignore timing precision, and omit false-positive measurement on safe scenes. We present PaSBench-Video, a 740-video benchmark with 481 risk and 259 no-risk videos across four domains: driving, healthcare, daily life, and industrial production. Risk videos are annotated with frame-level risk onset and accident boundaries. A model must observe the video causally and produce a warning that is both temporally calibrated and content-correct. Testing 13 MLLMs, we find that no model exceeds 20.0% on our strictest metric, and recall is tightly coupled with false-positive rate, with Pearson correlation 0.64: higher detection comes only at the cost of triggering warnings on the majority of safe clips. Performance splits sharply by domain: models achieve moderate recall at low false-positive rates in daily life, where risks are inherently anomalous, yet fire indiscriminately in driving, where routine and hazardous scenes look alike. These results indicate that current models rely on scene-level activity cues rather than reasoning about emerging harm.