EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation
EvDistill combines bidirectional reconstruction and cross-modal distillation, reaching 58.02% mIoU on unlabeled DDD17 events.
Key Findings
Methodology
EvDistill trains an image teacher T on labeled source images and an event student Sev on unlabeled, unpaired target events. Its Bidirectional Modality Reconstruction (BMR) uses GT→S and GS→T to create cross-modal supervision; Distribution Adaptation (DA) reduces modality shift; Affinity Graph KD transfers spatial relations; Mutual Distillation (MD) optionally links Sev and Saps. The objective is L=LCE+LBMR+λ1LDA+λ2LAG+λ3LMD.
Key Results
- On DDD17, multi-channel events obtain 58.02% mIoU at 50 ms, versus 54.81% for EvSegNet. Scores at 10 ms and 250 ms are 49.21% and 52.01%, approximately 7.5% and 9.5% above EvSegNet.
- The APS student reaches 72.63% mIoU, compared with 64.98% for pseudo-label-trained EvSegNet. On MVSEC, event and APS models reach 55.09% and 68.85%, improving over baselines by 8.8% and 11.2%.
- No event annotations are used for training. Multi-channel events outperform the two-channel voxel representation, 58.02% versus 57.16%, while qualitative results show robustness in HDR, low-light, and fast-motion scenes.
Significance
The paper addresses a central bottleneck in event vision: the scarcity of accurate pixel-level labels. It shows that a mature image model can transfer semantic capability to events without paired samples or event annotations. This combines event cameras’ HDR and low-blur advantages with the data ecosystem of conventional vision, offering a practical route for autonomous driving, robotics, and high-speed sensing.
Technical Contribution
BMR connects image and event domains through two generators and is optimized by task-aware Dynamic Semantic Consistency, KL[T(GT→S(e))||Sev(e)]. DA combines prediction matching with conditional adversarial modality alignment. Instead of regressing incompatible feature values, Affinity Graph KD matches local relations, Auv=FuᵀFv/(||Fu||2||Fv||2), across H×W nodes and σ-neighborhoods. The reconstruction module is removed after training, adding no inference cost.
Novelty
Unlike cross-modal KD methods requiring paired data, shared labels, APS pseudo-labels, or network grafting, EvDistill targets genuinely unpaired and unlabeled events. Its fundamental innovation is a task-driven bridge: reconstruction creates distillation pairs, distribution alignment handles domain shift, and affinity graphs transfer spatial structure. This is more modality-tolerant than direct feature matching and preserves a lightweight event-only inference path.
Limitations
- Evaluation is concentrated on DDD17, MVSEC, and E2VID. E2VID has no ground-truth segmentation labels, so its evidence is qualitative; generalization across sensors, environments, and tasks remains incompletely quantified.
- BMR introduces generators, discriminators, and adversarial optimization, making training more complex and potentially sensitive to event sparsity, noise, temporal windows, and reconstruction quality. The supplied paper text does not report full ablations, runtime, or energy costs.
Future Work
Useful directions include sensor-agnostic event representations, unsupervised cross-camera adaptation, and purely event-based training without APS. Future studies should report complete component ablations, latency, energy, and robustness, and extend the framework to detection, tracking, temporal transformers, and spiking neural networks.
AI Executive Summary
Event cameras report per-pixel brightness changes asynchronously rather than conventional frames. This gives them high dynamic range and little motion blur, but it also makes large-scale semantic annotation difficult. Existing approaches such as EvSegNet often derive pseudo-labels from low-quality APS frames, while reconstruction-based pipelines may require labels or add inference latency. The result is a mismatch between the hardware’s sensing advantages and the quality of its learned end-task models.
EvDistill treats labeled images as a source modality and unlabeled, unpaired events as a target modality. A teacher T learns from labeled images, while Sev learns from event representations. Bidirectional Modality Reconstruction uses GT→S and GS→T to form temporary cross-modal training pairs. Distribution Adaptation aligns predictions and modality statistics, Affinity Graph KD transfers spatial relations rather than raw heterogeneous features, and Mutual Distillation optionally couples event and APS students. BMR is removed at inference.
On DDD17, multi-channel events achieve 58.02% mIoU at 50 ms, compared with 54.81% for EvSegNet; scores at 10 ms and 250 ms are 49.21% and 52.01%. The APS student reaches 72.63% versus 64.98%. On MVSEC, event and APS results are 55.09% and 68.85%, above baselines of 50.53% and 61.93%. The study therefore demonstrates that image knowledge can supervise event-only learning without event labels or paired samples, while retaining event cameras’ strengths in HDR, low-light, and fast-motion conditions.
Deep Analysis
Background
Event cameras output asynchronous events (u,t,p), making them attractive for HDR and fast motion. Deep models have been applied to DDD17, N-Caltech, and MVSEC, but event labels remain sparse and costly. EvSegNet uses APS-derived pseudo-labels; Vid2E uses synthetic data; reconstruction systems such as E2Vid improve visual appearance but generally still require labels or increase inference cost.
Core Problem
The target is an event-only student trained without event annotations and without paired image-event samples. The challenge combines modality-specific appearance and statistics, sparse and noisy measurements, unreliable APS pseudo-labels, and the need to preserve spatial structure for pixel-wise segmentation. Standard feature KD assumes a shared modality and therefore cannot be transferred directly.
Innovation
- ��BMR uses GT→S and GS→T to construct cross-modal supervision and task-aware semantic consistency. •DA combines prediction matching with conditional adversarial alignment. •Affinity Graph KD transfers local spatial similarity instead of incompatible feature values. •MD lets event and APS students regularize each other. •BMR is training-only, so the final event model incurs no reconstruction overhead.
Methodology
- ��Input: encode event streams as multi-channel event images; use labeled XS={xs,ys} and unlabeled XT={e,xaps}. •Teacher/student: train T on labeled images, Sev on events, and optionally Saps on APS frames. •BMR: GT→S generates an image-like representation and GS→T generates an event-like representation; APS enables an L1 pixel loss, with cycle and adversarial objectives. •Distillation: use KL prediction matching, conditional DA, and AG loss. For feature map F, affinity is Auv=FuᵀFv/(||Fu||2||Fv||2). •Optimization: L=LCE+LBMR+λ1LDA+λ2LAG+λ3LMD; retain only the student at inference.
Experiments
Semantic segmentation is evaluated on DDD17, MVSEC, and E2VID using six classes and mIoU. DDD17 tests 10, 50, and 250 ms event windows and compares EvSegNet, Vid2E, voxel bins, and multi-channel representations. Event annotations are not used for training; available pseudo-labels are used only for test comparison. MVSEC includes outdoor day2 and night scenes. E2VID uses roughly 4K training and 400 test event images and is evaluated qualitatively.
Results
On DDD17, multi-channel events score 58.02% at 50 ms, versus 57.16% for two-channel voxel bins and 54.81% for EvSegNet; 10 ms and 250 ms scores are 49.21% and 52.01%. The APS result is 72.63% versus 64.98%. On MVSEC, events reach 55.09% and APS 68.85%, compared with 50.53% and 61.93% baselines. E2VID qualitative results show segmentation of vehicles, pedestrians, traffic lights, and complex structures.
Applications
The method directly targets autonomous driving under low light, overexposure, and rapid motion. It also suits robot navigation, drones, industrial inspection, and high-speed manipulation. Deployment requires an event sensor and a source-domain image teacher; APS can assist training when available, but the final system can run only Sev, preserving low-latency event inference.
Limitations & Outlook
Training depends on a source image teacher, reconstruction networks, discriminators, and adversarial optimization, so it is more complex than supervised learning. Performance may vary with sensor resolution, event noise, temporal windows, and domain shift. E2VID lacks quantitative ground truth, and the supplied text provides no detailed object-recognition numbers. Future work should test more sensors and tasks and measure latency, energy, and robustness systematically.
Plain Language Accessible to non-experts
Imagine two schools training a student to recognize road scenes. The first school has millions of clear photographs and answer sheets, so it can train an excellent teacher. The second school has only short messages saying that certain pixels changed brightness. These messages are powerful in darkness or during fast motion, but they do not come with labels such as “car” or “road,” and they do not line up with the photographs.
EvDistill hires two temporary translators. One turns change messages into something photograph-like; the other turns photographs into change-message-like information. The teacher and student can then compare answers. They also compare which nearby points belong together, like recognizing that many marks are parts of one car rather than unrelated dots. If an ordinary camera is available, a second student can check the event student’s answers.
After training, the translators and teacher leave. Only the fast event student remains. On DDD17 it reaches 58.02% mIoU, above the previous 54.81% result, showing that a system can learn useful visual meaning from unlabeled event data without reconstructing images during operation.
ELI14 Explained like you're 14
Think of an event camera as a gamer who does not record the whole screen. It only sends alerts when a pixel changes: “something moved here!” That sounds incomplete, but it is awesome for racing games, dark rooms, and sudden action because the alerts arrive quickly and do not blur.
The problem is homework: how can we teach the camera to say “car,” “road,” or “tree” when nobody has labeled millions of event alerts? EvDistill brings in a smart teacher trained on ordinary pictures. Temporary translators let the teacher and event student exchange clues, even though their data are not paired. The student also learns which nearby clues belong to the same object—like realizing that several dots are one racing car, not random noise.
When training ends, the translators disappear. The event student works alone and still makes predictions. On DDD17, it scores 58.02% mIoU, beating EvSegNet’s 54.81%. Why is that useful? In a dark driving scene or a super-fast game, the normal camera may fail, while event alerts still capture the important motion. Pretty clever, right?
Glossary
Event camera
A sensor that outputs an event (u,t,p) when brightness changes exceed a threshold, rather than recording regular frames. It provides HDR sensing and reduced motion blur.
It is the target-modality input for Sev.
Knowledge distillation
Training a student to reproduce a stronger teacher’s predictions or structural information. EvDistill uses KL divergence, affinity relations, and mutual distillation.
Image knowledge is transferred to event learning.
Bidirectional Modality Reconstruction
Two generators, GT→S and GS→T, create intermediate representations between image and event domains. The bridge is used only during training.
It creates usable supervision from unpaired modalities.
Distribution Adaptation
A procedure that reduces source-target shift by matching predictions and making feature distributions difficult to classify by modality. It includes LapsDA, LevDA, and LmatchDA.
It aligns image and event task representations.
Affinity Graph KD
Knowledge distillation based on pairwise spatial similarity among local feature nodes rather than direct feature-value regression. It uses normalized dot products between teacher and student affinities.
It transfers spatial structure for segmentation.
mIoU
The mean, across classes, of intersection-over-union between predicted and reference regions. Higher mIoU indicates more accurate semantic segmentation.
It is the main metric on DDD17 and MVSEC.
Open Questions Unanswered questions from this research
- 1 The paper does not fully isolate the gains from BMR, DA, and AG-KD, nor does the supplied text provide comprehensive sensitivity studies. More controlled ablations are needed.
- 2 E2VID has no ground-truth labels, and quantitative transfer across sensors, weather, resolutions, and end tasks remains unclear.
- 3 The energy, convergence stability, and wall-clock cost of generator-discriminator training are not systematically reported.
Applications
Immediate Applications
Low-light autonomous driving
A vehicle can use event streams for road, vehicle, and vegetation segmentation in darkness, backlight, or overexposure. Deployment requires an event camera and a pretrained image teacher; unlabeled driving events suffice for target training.
High-speed robotic vision
Robots can use Sev to detect fast objects and fine structures with reduced blur. After distillation, only the event student runs, making the approach suitable for low-latency navigation, grasping, and industrial inspection.
Long-term Vision
Sensor-agnostic perception
Combining BMR with temporal models or spiking networks could let heterogeneous event sensors share image-derived semantics. This may support all-weather autonomous vehicles and drones, but needs stronger cross-domain validation.
Abstract
Event cameras sense per-pixel intensity changes and produce asynchronous event streams with high dynamic range and less motion blur, showing advantages over conventional cameras. A hurdle of training event-based models is the lack of large qualitative labeled data. Prior works learning end-tasks mostly rely on labeled or pseudo-labeled datasets obtained from the active pixel sensor (APS) frames; however, such datasets' quality is far from rivaling those based on the canonical images. In this paper, we propose a novel approach, called \textbf{EvDistill}, to learn a student network on the unlabeled and unpaired event data (target modality) via knowledge distillation (KD) from a teacher network trained with large-scale, labeled image data (source modality). To enable KD across the unpaired modalities, we first propose a bidirectional modality reconstruction (BMR) module to bridge both modalities and simultaneously exploit them to distill knowledge via the crafted pairs, causing no extra computation in the inference. The BMR is improved by the end-tasks and KD losses in an end-to-end manner. Second, we leverage the structural similarities of both modalities and adapt the knowledge by matching their distributions. Moreover, as most prior feature KD methods are uni-modality and less applicable to our problem, we propose to leverage an affinity graph KD loss to boost the distillation. Our extensive experiments on semantic segmentation and object recognition demonstrate that EvDistill achieves significantly better results than the prior works and KD with only events and APS frames.