Pedestrian Crossing Intent Classification From Event-Based Vision Using Convolutional Spiking Neural Networks With Temporal Augmentation
Classifying pedestrian crossing intent using Conv-SNN with temporal augmentation, achieving 95.83% accuracy.
Key Findings
Methodology
The study presents an end-to-end pipeline that converts driving footage from the JAAD dataset into DVS event streams using the v2e simulator and augments training with CARLA-simulated DVS-PedX sequences. A novel Conv-SNN is trained with leaky-integrate-and-fire neurons and surrogate-gradient learning, using class-balanced loss and 6x JAAD oversampling.
Key Results
- The model achieved 95.83% accuracy and an F1 score of 0.9695 on the JAAD DVS test set, demonstrating high efficiency in real driving scenarios.
- On normal CARLA DVS, it achieved 97.79% accuracy and an F1 score of 0.9478, proving robustness in simulated environments.
- On adverse-weather CARLA DVS, it achieved 94.78% accuracy and an F1 score of 0.8369, indicating adaptability under challenging conditions.
Significance
This research is significant for autonomous driving as it enhances pedestrian crossing intent prediction accuracy, improving vehicle safety. Compared to traditional RGB video methods, it offers significant advantages in temporal resolution and energy efficiency, especially in handling motion blur and high dynamic range scenarios.
Technical Contribution
The study's technical contributions include the first application of Conv-SNN to event-based pedestrian intent prediction, providing new theoretical guarantees and engineering possibilities. The introduction of temporally consistent DVS augmentation addresses previous methods' shortcomings in temporal coherence.
Novelty
This method is the first to convert JAAD dataset RGB videos into DVS event streams and apply Conv-SNN for intent classification, bridging the gap between event-based perception and SNN classifiers in pedestrian intent prediction.
Limitations
- The method may experience performance degradation in highly complex scenes due to the model's large parameter size, which could increase computational overhead.
- In extreme weather conditions, DVS sensors may not capture enough events, affecting prediction accuracy.
Future Work
Future research could explore training on larger datasets to enhance model generalization. Additionally, real-time deployment on embedded platforms could be studied to verify performance in practical applications.
AI Executive Summary
In the field of autonomous driving, predicting whether a pedestrian will cross the street is crucial for vehicle safety. However, traditional frame-based RGB video methods struggle with motion blur and high dynamic range scenarios.
This study proposes a novel approach using Convolutional Spiking Neural Networks (Conv-SNN) and temporal augmentation. By converting driving footage from the JAAD dataset into Dynamic Vision Sensor (DVS) event streams and augmenting with CARLA-simulated DVS-PedX sequences, the method achieves efficient classification of pedestrian crossing intent.
Experimental results show that the method achieves 95.83% accuracy on the JAAD DVS test set and performs well under various weather conditions in CARLA DVS, demonstrating its potential for real-world driving scenarios. Despite this, the method's performance in extremely complex scenes requires further validation.
Deep Analysis
Background
The development of autonomous driving technology requires accurate prediction of pedestrian intent. Traditional methods primarily rely on RGB video data but face limitations in dynamic scenes. Recently, Dynamic Vision Sensors (DVS) have gained attention for their high temporal resolution and low energy consumption.
Core Problem
The core problem is accurately predicting whether a pedestrian will cross the street in real-time. Traditional methods perform poorly in motion blur and high dynamic range scenarios and have high computational costs.
Innovation
The core innovations of this study include applying Conv-SNN to event-based data and improving prediction accuracy and robustness through temporally consistent DVS augmentation.
Methodology
- �� Convert JAAD dataset videos to DVS event streams
- �� Train with CARLA-simulated DVS-PedX sequences
- �� Use Conv-SNN for classification
- �� Apply class-balanced loss and 6x oversampling
Experiments
Experiments use the JAAD and CARLA DVS-PedX datasets with a 70/15/15 stratified splitting protocol for training and testing. The model is trained on a CPU with 1.07M parameters.
Results
The model achieved 95.83% accuracy on the JAAD DVS test set. On normal and adverse-weather CARLA DVS, it achieved 97.79% and 94.78% accuracy, respectively.
Applications
This method can be directly applied to pedestrian intent prediction for autonomous vehicles, particularly suitable for dynamic scenes and complex weather conditions.
Limitations & Outlook
The model may experience performance degradation in extremely complex scenes, and computational overhead is high. Future work should optimize the model for embedded platforms.
Plain Language Accessible to non-experts
Imagine a smart traffic system like a super-intelligent traffic light. Traditional traffic lights change based on preset times, but this smart light can observe pedestrians in real-time and decide if they will cross the street. It uses a special 'eye' that sees rapid changes in detail, unaffected by light changes or motion blur. This 'eye' is the Dynamic Vision Sensor (DVS). Its brain is a clever neural network that quickly makes decisions. This system not only improves traffic efficiency but also significantly enhances pedestrian safety.
ELI14 Explained like you're 14
Imagine you're playing a super cool racing game where the car can drive itself! But it needs to know when someone wants to cross the street. Traditional methods are like using an old camera, slow and blurry. The new method is like giving the car a super-fast camera that sees every tiny movement change. Then, the smart system inside the car acts like a super-smart assistant, quickly deciding if a pedestrian wants to cross, making the car stop safely. Isn't that amazing?
Glossary
Dynamic Vision Sensor
A sensor that detects per-pixel brightness changes, offering high temporal resolution and low energy consumption.
Used to generate event streams as an alternative to traditional RGB video.
Convolutional Spiking Neural Network
A network structure combining convolutional layers and spiking neurons, mimicking the communication mechanism of biological neurons.
Used to process DVS event streams and classify intent.
Leaky-Integrate-and-Fire Neuron
A neuron model that accumulates input current and fires a spike when a threshold is reached.
Used in Conv-SNN to process temporal information.
Surrogate Gradient Learning
A technique for training spiking neural networks by using a smooth approximation to replace the non-differentiable spike function.
Used to optimize the training process of Conv-SNN.
Temporal Augmentation
A data augmentation technique that applies consistent temporal transformations to improve model robustness.
Used in training DVS data to maintain temporal consistency.
Open Questions Unanswered questions from this research
- 1 How to improve prediction accuracy in extremely complex scenes? Current methods perform poorly with complex backgrounds and multiple pedestrian interactions, requiring more efficient feature extraction techniques.
- 2 How to achieve real-time deployment on embedded platforms? Current models have high computational overhead, needing optimization for resource-constrained environments.
Applications
Immediate Applications
Autonomous Vehicles
Enhance pedestrian crossing intent prediction accuracy, improving vehicle safety and response speed.
Smart Traffic Systems
Improve traffic light intelligence by monitoring pedestrian dynamics in real-time.
Long-term Vision
Smart Cities
Integrate advanced vision sensing technology to enhance urban traffic management's intelligence and automation.
Abstract
Anticipating whether a pedestrian will cross the road is safety-critical for autonomous vehicles, requiring real-time inference under challenging conditions including motion blur, high dynamic range, and class imbalance. Conventional frame-based deep networks process redundant RGB data at fixed frame rates, limiting their temporal resolution and energy efficiency. In this work we present an end-to-end pipeline that (i) converts real-world driving footage from the Joint Attention in Autonomous Driving (JAAD) dataset into synthetic dynamic vision sensor (DVS) event streams using the v2e simulator, (ii) augments training with the CARLA-simulated DVS sequences of the DVS-PedX dataset under both normal and adverse weather conditions, and (iii) trains a novel convolutional spiking neural network (Conv-SNN) with clip-consistent DVS augmentation to classify pedestrian crossing intent as binary: crossing or non-crossing. We detail all architectural decisions, the exact leaky-integrate-and-fire neuron dynamics with surrogate-gradient learning, the class-balanced loss formulation, JAAD oversampling at 6x, and a 70/15/15 stratified splitting protocol. The trained model achieves 95.83% accuracy and F1 = 0.9695 on the JAAD DVS test set, 97.79% accuracy and F1 = 0.9478 on normal CARLA DVS, and 94.78% accuracy and F1 = 0.8369 on adverse-weather CARLA DVS, all from a 1.07M-parameter architecture trained on CPU. Compared to prior frame-based approaches on JAAD, our method closes or surpasses the reported accuracy while operating natively on sparse temporal representations. We include a thorough analysis of the convergence behaviour across all 15 training epochs, domain transfer characteristics, and a quantitative comparison with representative related work.