CNN based Road User Detection using the 3D Radar Cube
RTCnet fuses target features with local 3D radar-cube data, achieving 0.70 target-level and 0.56 object-level F1.
Key Findings
Methodology
The paper introduces RTCnet, or Radar Target Classification Network. Each dynamic radar target is mapped to the 3D radar cube and surrounded by a cropped L=W=5, H=32 block. Two 3D-convolution stages encode range–azimuth context and Doppler structure; three 1D convolutions then process the Doppler axis. The resulting representation is concatenated with (r,α,vr,RCS) and classified by two 128-unit fully connected layers. Class-specific DBSCAN subsequently produces object proposals.
Key Results
- On real urban-road data, RTCnet reaches a macro-average target-wise F1 of 0.70, versus 0.68 for Schumann and 0.61 for Prophet. Per-class F1 is 0.71 for pedestrians, 0.67 for cyclists, and 0.50 for cars.
- Object-level evaluation defines IoU by radar-target counts and uses IoU≥0.5 as a true positive. RTCnet obtains average F1=0.56, compared with 0.48 for Schumann; pedestrian and car F1 are 0.61 and 0.47.
- Ablation confirms the value of low-level data: removing the radar cube reduces target F1 from 0.70 to 0.61. Removing RCS gives 0.69, while removing ensemble voting gives 0.67.
Significance
The work addresses a central automotive-radar trade-off: radar targets offer relatively accurate positions but are sparse, whereas the radar cube contains richer Doppler structure at lower spatial resolution. RTCnet delivers target-level labels and object proposals from one sweep, approximately 75 ms, making it relevant to low-latency urban driving and to intermediate- and late-stage sensor fusion.
Technical Contribution
Instead of clustering first and classifying clusters, the system fuses engineered target features with a local crop from the original 3D cube. 3D convolutions compress range and azimuth while preserving Doppler information; 1D convolutions specialize in velocity-distribution patterns. Predicted class scores then guide class-specific DBSCAN. Different classes can use different spatial, velocity, and MinPoints settings, followed by cluster merging using position, velocity, and normalized class-score distances.
Novelty
Compared with Prophet and Schumann, RTCnet classifies individual targets before segmentation. Compared with methods based on Range-Doppler projections or long Doppler-Time windows, it uses a single-frame local crop of the full 3D cube while retaining accurate target localization. Its fundamental novelty is closing the loop: classification is used to make clustering class-aware rather than treating clustering as a fixed preprocessing step.
Limitations
- Specular reflections, ghost targets, nearby strong returns, unusual vehicles, and partial misclassification of one vehicle’s returns can generate fragmented or duplicated object proposals.
- The study uses about one hour of data from one Continental 400-series radar and two independent routes, so robustness across sensors, weather, cities, and class distributions is not established.
- Labels were generated with SSD and SGM stereo processing before manual correction; projection and depth errors can still introduce radar–vision annotation mismatch.
Future Work
Future research should test cross-sensor and cross-domain transfer, adverse-weather robustness, and efficient temporal aggregation without sacrificing single-sweep latency. Sparse 3D networks, spatiotemporal attention, calibrated uncertainty, and joint tracking could reduce ghost-induced errors and improve downstream planning. Larger datasets should quantify generalization across vehicles, radar configurations, and geographic environments.
AI Executive Summary
Automotive radar remains attractive because it works in darkness, rain, and snow while measuring radial velocity directly through the Doppler effect. Yet conventional road-user detectors often apply DBSCAN to sparse radar targets first and classify each resulting cluster. This pipeline is brittle: nearby objects may be merged, large vehicles may be split, and single-return pedestrians may be discarded when MinPoints exceeds one. Once clustering fails, classification cannot recover.
Palffy and colleagues propose RTCnet, a single-frame detector for pedestrians, cyclists, and cars. The system maps each dynamic target into a 3D radar cube and crops a 5×5×32 local block across range, azimuth, and Doppler dimensions. 3D convolutions encode spatial context, 1D convolutions analyze local velocity structure, and two 128-unit fully connected layers combine these features with range, azimuth, compensated speed, and RCS. Ten One-vs-All and One-vs-One binary classifiers vote on target classes; class-specific DBSCAN then forms object proposals.
The real-world dataset contains 31,300 pedestrian, 15,290 cyclist, and 9,362 car instances, with many single-return objects. RTCnet achieves target-wise macro F1=0.70, above Schumann’s 0.68 and Prophet’s 0.61, and object-wise F1=0.56 versus 0.48. Removing low-level cube information drops target F1 to 0.61, demonstrating that local Doppler structure carries discriminative motion cues. Inference takes about 0.04 s on a TITAN V system, although ghost reflections, unusual vehicles, and cross-sensor generalization remain important unresolved issues.
Deep Analysis
Background
Radar is robust to illumination and weather and provides direct radial velocity. Prophet used DBSCAN plus Random Forest for pedestrian detection; Schumann extended cluster-wise classification to multiple classes but relied on cluster features and multiple frames. Angelov used CNN-LSTM models over Range-Doppler and Doppler-Time spectrograms. The 3D cube preserves range, azimuth, and Doppler structure, including motion signatures from limbs and wheels, but has lower spatial resolution than processed radar targets.
Core Problem
A single radar sweep is sparse and irregular. DBSCAN can merge neighboring road users or split one large car into several clusters. One parameter set cannot naturally fit pedestrians and cars, while MinPoints>1 discards single-return objects. Conversely, target features (r,α,vr,RCS) lack rich local motion shape, whereas raw cube data is spatially coarse. The challenge is to combine precise localization with informative Doppler context under urban-driving latency constraints.
Innovation
First, RTCnet classifies targets before clustering, allowing single-return measurements to receive class information. Second, it uses a local crop from the full 3D cube rather than only a 2D Range-Doppler projection. Third, predicted classes enable class-specific DBSCAN: the reported spatial thresholds are 0.5 m for pedestrians, 1.6 m for cyclists, and 4.0 m for cars. Fourth, spatial, velocity, and class-score distributions are used to merge clusters that were split by occasional misclassification.
Methodology
- ��Preprocessing: compensate ego motion, remove targets with absolute compensated speed below 0.3 m/s, and map (r,α,vr) to cube bins.
- ��Local representation: crop L=W=5 and H=32 bins around each target; normalize r, α, vr, RCS, and the cube using training statistics.
- ��Spatial encoding: two 3D convolutions with output channels 6 and 25, followed by max-pooling, compress range and azimuth while retaining Doppler structure.
- ��Doppler processing: three 1D convolutions along Doppler with channels 16, 32, and 32; each is followed by pooling with kernel 3 and stride 2, yielding a 32×1×1×H/8 tensor.
- ��Scoring: flatten the tensor, concatenate (r,α,vr,RCS), and use two 128-unit fully connected layers. The output has four multi-class nodes or two binary nodes.
- ��Proposal generation: combine 10 OvA/OvO classifiers by voting, run class-specific DBSCAN, then merge nearby clusters using position, velocity, and class-score distances.
Experiments
The dataset contains about one hour of urban driving using a Continental 400-series radar behind the bumper, a 1936×1216 stereo camera, and odometry. SSD trained on EuroCity Persons supplied image detections; SGM estimated stereo distance, followed by manual corrections. The training set includes 31,300 pedestrians, 15,290 cyclists, and 9,362 cars. Independent driving sessions provided training and testing separation. Baselines were reimplemented Prophet and Schumann with DBSCAN and 50-tree Random Forests. Metrics were target-wise F1 and object-wise F1, with object IoU computed over radar-target counts and IoU≥0.5 defining a true positive. Models trained for 10 epochs.
Results
RTCnet target-wise F1 is 0.71 for pedestrians, 0.67 for cyclists, and 0.50 for cars, averaging 0.70. Schumann scores 0.67, 0.68, and 0.46, averaging 0.68; Prophet averages 0.61. Object-wise average F1 is 0.56 versus Schumann’s 0.48, with car F1 improving from 0.31 to 0.47 and pedestrian F1 from 0.54 to 0.61. Ablations yield 0.61 without low-level data, 0.64 without speed, 0.69 without RCS, and 0.67 without ensembling. Inference is approximately 0.04 s.
Applications
The approach can support adaptive cruise control, automatic emergency braking, urban-intersection monitoring, and night-time pedestrian protection. Target-level class scores are useful for measurement-level sensor fusion, while object proposals can feed tracking, trajectory prediction, and planning. Deployment requires a radar exposing cube data, reliable ego-motion estimates, target extraction, and sensor-specific calibration of DBSCAN and merging thresholds.
Limitations & Outlook
The method filters static targets and relies on one sweep, so it cannot exploit temporal consistency to disambiguate weak or unstable returns. Specular surfaces and ghost reflections can resemble genuine road users, while unusual vehicles may fall outside the training distribution. Class-aware clustering is flexible but can create duplicate objects when some returns are misclassified. The dataset’s concentrated sensor and route coverage limits claims of generalization. Future work should combine efficient temporal models, domain adaptation, sparse 3D architectures, uncertainty calibration, and joint detection–tracking evaluation.
Plain Language Accessible to non-experts
Imagine a worker sorting packages at a busy delivery center during a power outage. The worker cannot read colorful labels, but a special scanner reports where each package is, how fast it is moving, and how strongly it reflects the scanner’s signal. Older sorting systems first group nearby packages into boxes and then decide what each box contains. If two customers’ packages are mixed, or one customer’s large shipment is split into several boxes, the final decision becomes unreliable.
RTCnet changes the order. It examines each package first, then looks at a small neighborhood around it. That neighborhood records not only location but also a whole pattern of movement. A person’s swinging arms, a bicycle’s rotating wheels, and a car’s larger body can create different patterns. The system learns these patterns and combines them with the package’s basic scanner information.
Only afterward does it group items into objects. It uses stricter grouping for small objects and wider grouping for cars, then checks whether nearby groups probably belong together. In testing, its average item-level F1 was 0.70 and object-level F1 was 0.56, better than the comparison results of 0.68 and 0.48. Removing the neighborhood movement map reduced performance to 0.61, showing that the extra local information matters.
ELI14 Explained like you're 14
Think of a self-driving car playing a real-world racing game in thick fog. Its camera is struggling, but radar still says, “Something is 12 meters away, moving this fast, in that direction.” The tricky part is that a person, bicycle, or car may produce only a few dots. Two nearby objects can also make their dots look like one messy group.
RTCnet acts like a clever teammate who checks every dot before making groups. It looks at a tiny 3D neighborhood around the dot, including how radar signals are spread across different speeds. Why is that useful? A walking person moves arms and legs, a bicycle has spinning wheels, and a car usually produces a different, broader pattern. The AI combines this clue with distance, direction, speed, and reflection strength.
Then it groups dots that probably belong to the same road user. It does not use exactly the same rule for everyone: cars are big, so their dots may be farther apart; pedestrians are small, so grouping must be careful. That is like using different team-formation rules for a bus and a child.
The results are strong: average target F1 is 0.70, and object F1 is 0.56, beating comparison scores of 0.68 and 0.48. Without the local speed map, performance falls to 0.61. Still, shiny walls, strange vehicles, and radar “ghosts” can fool it. Future versions could watch several frames and say how confident they are!
Glossary
3D radar cube
A three-dimensional array indexed by range, azimuth, and Doppler velocity, where each cell stores radar reflectivity. It preserves local motion distributions that processed target points may omit.
RTCnet crops a 5×5×32 block around every target.
Doppler
A frequency shift used to estimate radial motion. Multiple moving body parts can create structured patterns along the Doppler axis.
RTCnet uses 1D convolutions to analyze this axis.
RTCnet
Radar Target Classification Network, a CNN that classifies individual radar targets using cube crops and target-level features. It provides scores before object clustering.
It predicts pedestrian, cyclist, car, and other classes.
DBSCAN
A density-based unsupervised clustering algorithm that forms groups from neighborhood distance and density without specifying the number of clusters. Its parameters control spatial and velocity association.
The paper applies it separately to predicted classes.
F1 score
The harmonic mean of precision and recall: F1=2PR/(P+R). It balances missed detections and false positives.
RTCnet achieves 0.70 target-wise and 0.56 object-wise average F1.
RCS
Radar cross section, a measure of how strongly an object reflects radar energy. It provides physical cues but varies with geometry and aspect angle.
RCS is one of the target-level inputs; removing it gives F1=0.69.
Open Questions Unanswered questions from this research
- 1 Cross-sensor generalization remains unclear: the study uses a Continental 400-series radar, so performance across radar manufacturers, bands, and resolutions needs direct evaluation.
- 2 It is unresolved how to exploit multiple frames without losing the approximately 75 ms latency required for urban driving; efficient temporal models are needed.
- 3 Ghost reflections and unusual vehicles lack explicit uncertainty handling, requiring broader annotations, calibration, and safety-oriented evaluation.
Applications
Immediate Applications
Night-time pedestrian warning
Vehicle systems can use RTCnet pedestrian scores to complement cameras in darkness, rain, or snow. Required inputs are extracted radar targets, a 3D radar cube, and ego-motion estimates. The reported implementation processes a frame in about 0.04 s on a TITAN V platform.
Urban-intersection perception
Object proposals can be passed to trackers and planners while class-specific DBSCAN reduces merging between pedestrians, cyclists, and cars. Deployment requires recalibration of spatial, velocity, and cluster-merging thresholds for each radar installation.
Long-term Vision
Radar-centered all-weather autonomy
With temporal tracking and camera/LiDAR fusion, local Doppler modeling could become a core all-weather perception component. Major obstacles include domain shift, sensor variation, ghost suppression, large-scale data collection, and automotive safety certification.
Abstract
This letter presents a novel radar based, single-frame, multi-class detection method for moving road users (pedestrian, cyclist, car), which utilizes low-level radar cube data. The method provides class information both on the radar target- and object-level. Radar targets are classified individually after extending the target features with a cropped block of the 3D radar cube around their positions, thereby capturing the motion of moving parts in the local velocity distribution. A Convolutional Neural Network (CNN) is proposed for this classification step. Afterwards, object proposals are generated with a clustering step, which not only considers the radar targets' positions and velocities, but their calculated class scores as well. In experiments on a real-life dataset we demonstrate that our method outperforms the state-of-the-art methods both target- and object-wise by reaching an average of 0.70 (baseline: 0.68) target-wise and 0.56 (baseline: 0.48) object-wise F1 score. Furthermore, we examine the importance of the used features in an ablation study.