STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision
STAR-VLM uses nuScenes radar supervision to train VLMs, reaching 0.94 motion accuracy and 1.20 m/s radial-velocity MAE.
Key Findings
Methodology
STAR-VLM fine-tunes a 4B Qwen3-VL model with radar-derived radial velocity, motion state, and ego-velocity. Radar points are projected into camera images, while joint pixel coordinates and red-arrow visual prompts identify queried locations. STAR-Bench-radar supplies question-answer supervision; standard supervised fine-tuning uses next-token prediction and cross-entropy, without architectural changes or a dedicated regression head.
Key Results
- On STAR-Bench, STAR-VLM reaches 0.80 motion-classification accuracy against human annotations and 0.94 against radar labels, exceeding Qwen3-VL 8B at 0.54 and 0.62.
- Radial-velocity MAE is 1.94 m/s on annotation labels and 1.37 m/s on radar labels; with joint ego-velocity training it becomes 1.89 and 1.20 m/s, outperforming Any4D at 2.31 and 3.60 m/s.
- On public TADBench, STAR-VLM obtains 0.81 VQA and 0.72 MCQ accuracy. Ablations show that motion-plus-radial-velocity training alone degrades performance, whereas adding ego-velocity restores accuracy to 0.80.
Significance
The paper shows that inexpensive, widely deployed automotive radar can act not only as an inference sensor but also as a training teacher for VLMs. Doppler measurements provide real-world velocity scale, addressing the gap between describing motion and estimating it quantitatively. Academically, the work connects cross-modal supervision, language reasoning, and metric temporal learning. Industrially, it reduces dependence on costly human labels and synthetic data for auto-labeling and end-to-end driving systems.
Technical Contribution
The main contributions are: converting radar range and Doppler measurements into pixel-level QA supervision; introducing joint textual-coordinate and visual-arrow prompting for stronger pixel grounding; and jointly training radial velocity, motion classification, and ego-velocity. Without modifying the VLM architecture, STAR-VLM uses next-token prediction and cross-entropy to acquire metric temporal capability, outperforming task-specific SegAnyMo and Any4D on their corresponding evaluation tasks.
Novelty
Unlike DepthLM, which primarily learns metric depth, STAR-VLM systematically uses automotive-radar Doppler supervision for metric temporal reasoning and object radial-velocity estimation. Its fundamental novelty is an efficient supervision strategy rather than a new network architecture: naturally available sensor signals become scalable language targets, while ego-velocity learning helps disentangle object motion from platform motion.
Limitations
- Radar measures mainly radial velocity and cannot directly recover full tangential motion; sparse returns, occlusion, multipath effects, and noisy associations can corrupt labels.
- Experiments focus on nuScenes front-view data and a 4B Qwen3-VL model, so generalization across cameras, weather, cities, radar suppliers, and larger models remains uncertain.
- Numerical answers are generated as text, making precision sensitive to prompting, rounding, and output-format errors.
Future Work
Future research should combine multi-view radar, cameras, and LiDAR to estimate full 3D velocity rather than only radial components. Important directions include calibrated uncertainty, online domain adaptation, broader cross-city evaluation, robust numerical decoding, and physical consistency constraints. Radar supervision could also extend to tracking, scene flow, planning, and vision-language-action models.
AI Executive Summary
Vision-language models can describe a driving scene, yet often cannot answer a physically crucial question: how many metres per second is a moving vehicle travelling? Existing video methods depend on costly annotation, elaborate preprocessing, or synthetic data. DepthLM improves metric depth but does not provide temporal motion supervision. STAR-VLM addresses this gap by using automotive radar as a scalable teacher, exploiting range and Doppler measurements collected in real driving.
The authors build STAR-Bench from nuScenes, using about 34K training images and roughly 900K radar points. Projected radar points are marked with a red arrow and specified by pixel coordinates; the model answers questions about motion state, radial velocity, and ego-velocity. Starting from 4B Qwen3-VL, STAR-VLM uses ordinary supervised fine-tuning and no architectural modification. Ego-velocity supervision helps distinguish target motion from motion caused by the host vehicle.
The results are substantial. Motion accuracy reaches 0.80 with annotation labels and 0.94 with radar labels; radial-velocity MAE is 1.94 and 1.37 m/s, improving to 1.89 and 1.20 m/s with ego-velocity training. On TADBench, it obtains 0.81 VQA and 0.72 MCQ accuracy, outperforming general VLMs, Cosmos-Reason, SegAnyMo, and Any4D. The study demonstrates that inexpensive radar can move VLMs from recognizing motion to measuring it, while full 3D velocity, cross-domain robustness, and radar noise remain open challenges.
Deep Analysis
Background
VLMs have progressed from static understanding to driving systems such as DriveLM, DriveVLM, and DriveMLM. STSBench and TADBench evaluate temporal reasoning, while SpatialVLM improves grounding. DepthLM shows that sensor supervision can enhance metric depth, but dynamic metric velocity remains underexplored. Automotive radar is attractive because it is low-cost, widely deployed, relatively weather robust, and directly measures range and Doppler.
Core Problem
The model must decide whether a queried object is moving relative to the camera and estimate its radial velocity in m/s while accounting for ego motion. Pure vision suffers from scale ambiguity; human velocity labels are expensive, and synthetic supervision may create sim-to-real gaps. Radar is physically informative but sparse, and penetrating returns can project onto image regions that are visually occluded.
Innovation
- �� STAR-Bench-radar converts radar range, Doppler, and motion state into QA supervision.
- �� STAR-Bench-anno projects human 3D-box labels for comparison.
- �� An occlusion filter removes points with a nearer neighbour within N=30 pixels and depth difference greater than τ=0.3 m.
- �� Joint textual coordinates and red arrows improve pixel grounding.
- �� Cross-task training combines radial velocity, motion classification, and ego-velocity to disentangle platform and object motion.
Methodology
- �� Inputs: nuScenes front camera images, radar points, and ego-pose signals.
- �� Projection: radar points are mapped to the image; likely occluded points are filtered using local image coordinates and depth.
- �� Querying: a red arrow marks the target pixel, while the prompt also provides (x,y).
- �� Outputs: the model generates motion state or radial velocity rounded to two decimals.
- �� Training: 4B Qwen3-VL is optimized with SFT, next-token prediction, and cross-entropy; ego-velocity QA supplies additional metric supervision.
- �� Evaluation: comparisons include Qwen2.5-VL, Qwen3-VL, Cosmos-Reason, DepthLM, SegAnyMo, and Any4D.
Experiments
Training uses the non-mini nuScenes trainval split: about 34K images and 900K radar points. The mini split contains about 400 evaluation images. Models train on four A100 GPUs for 12–30 hours. Metrics are motion-classification accuracy, radial-velocity MAE, and TADBench VQA/MCQ accuracy. Experiments compare radar and annotation supervision, prompting formats, base models, and cross-task ablations.
Results
STAR-VLM obtains 0.80 and 0.94 motion accuracy on annotation and radar labels, versus 0.54 and 0.62 for Qwen3-VL 8B. Its velocity MAE is 1.94 and 1.37 m/s, better than Any4D at 2.31 and 3.60 m/s. TADBench results are 0.81 VQA and 0.72 MCQ. Combining motion and radial velocity alone hurts performance to 0.62/0.81, while adding ego-velocity restores it to 0.80/0.94 and improves velocity error.
Applications
The method can support autonomous-driving auto-labeling by generating motion-state and radial-velocity labels from camera, radar, and ego-pose data, reducing manual annotation. It may also assist tracking, collision-risk analysis, and VLA training. Prerequisites include reliable calibration, synchronization, pose estimation, and a query mechanism; inference need not require radar if the model has learned radar supervision.
Limitations & Outlook
Radar provides only the line-of-sight velocity component, so complete 3D motion requires complementary cues. Occlusion filtering may discard valid returns or retain multipath noise. The benchmark is concentrated on nuScenes front-view driving, with limited evidence for severe weather, other cities, sensors, or domains. Training requires four A100 GPUs, and textual numerical generation can introduce formatting errors. Future work should add multi-view and tangential-motion supervision, uncertainty estimates, and physical constraints.
Plain Language Accessible to non-experts
Imagine a student driver who watches road video. The student can say, “That car is moving,” but cannot reliably tell whether it travels three or ten metres per second, because pictures do not directly reveal real-world scale. STAR-VLM gives the student a low-cost radar coach. The radar reports how far an object is and whether it is approaching or receding.
During training, researchers place each radar reading back onto the camera image, draw a red arrow there, and ask: “Is the object here moving, and how fast?” The model answers in words. It also learns the speed of its own car, because a changing image may result from the camera moving rather than the other car moving.
The results resemble a strong exam score: motion accuracy reaches 94%, and average radial-speed error falls to 1.20 m/s after ego-velocity training. The model therefore does more than notice change; it connects change to physical units. However, the radar coach mainly knows motion toward or away from itself. Sideways motion, hidden objects, and noisy echoes still require additional sensors and better reasoning.
ELI14 Explained like you're 14
Think about a racing game. You can look at the screen and tell that the car ahead is moving, but if the game removes the speedometer, can you accurately say how many metres per second it travels? A camera has the same problem: it sees changing pictures but does not automatically know real distance or speed.
STAR-VLM gives the AI a radar teammate, a bit like a bat using echoes. Radar measures how far an object is and whether it is coming closer or moving away. Researchers point a red arrow at a spot in the picture and ask the AI, “Is the thing here moving, and what is its speed?”
The AI must also know whether its own car is moving. Otherwise, a vehicle may appear to change position simply because the camera is travelling forward. Learning ego-velocity helps the AI separate the two effects.
In testing, STAR-VLM reaches 94% motion accuracy and about 1.20 m/s average radial-speed error. Pretty impressive! But it is still better at motion toward or away from the car than sideways motion. Rain, blocked objects, radar glitches, and confusing echoes can also fool it, so future systems will need multiple sensors.
Glossary
Vision-Language Model
A model that processes images and language and produces language-based reasoning or answers. Technically, it aligns visual representations with token-based language generation.
STAR-VLM fine-tunes Qwen3-VL without changing its architecture.
Doppler velocity
Velocity inferred from the frequency shift of radar echoes. It primarily measures motion along the sensor’s line of sight.
The paper uses Doppler measurements as label-free temporal supervision.
Radial velocity
The projection of an object’s velocity onto the sensor-to-object direction, not the complete 3D velocity. The paper evaluates it with mean absolute error.
STAR-VLM reaches 1.37 m/s radial-velocity MAE on radar labels.
Ego-velocity
The motion velocity of the vehicle carrying the sensors. It helps separate platform-induced image changes from genuine object motion.
Joint ego-velocity training improves both classification and velocity estimation.
STAR-Bench
A nuScenes-based spatiotemporal QA benchmark with radar-derived and annotation-derived variants. It evaluates motion classification and velocity estimation.
The radar variant is the main training source, while the mini split is used for evaluation.
Supervised Fine-Tuning
Continued training of a pretrained model on input-answer examples. In this paper, text-token cross-entropy replaces a specialized numerical regression head.
The model learns to output motion labels and rounded velocity values as text.
Open Questions Unanswered questions from this research
- 1 Radar mainly supplies radial velocity. A robust, unified method for combining it with visual motion and multi-view geometry to recover full 3D velocity remains unresolved.
- 2 STAR-VLM is evaluated mainly on nuScenes. Its robustness across cities, weather, camera configurations, radar suppliers, and operational domains is still unknown.
- 3 Text-generated numbers may depend on prompts and formatting. Calibrated uncertainty, reliable numerical decoding, and physical consistency need further study.
Applications
Immediate Applications
Autonomous-driving auto-labeling
Engineering teams can combine camera images, radar, and ego-pose to generate motion-state and radial-velocity labels, reducing manual annotation. Reliable calibration, synchronization, and occlusion filtering are required; the labels can support tracking, risk analysis, and model training.
Interpretable driving-scene analysis
A monitoring or debugging system can query an image location and receive a textual answer about whether the object moves and how fast it approaches or recedes. This is useful for readable auto-labels, dataset audits, and perception diagnostics.
Long-term Vision
Metric-aware driving VLA
Velocity-aware visual-language-action systems could use distance, motion, and ego-motion to make safer planning decisions rather than merely describing traffic. Major obstacles are complete 3D motion, domain robustness, uncertainty calibration, and reliable integration with control.
Abstract
Vision-language models (VLMs) are emerging as a key component of embodied intelligence, with growing applications in auto-labeling and end-to-end autonomous driving. However, existing approaches for improving spatiotemporal reasoning in VLMs often rely on complex preprocessing pipelines, expensive human annotations, or synthetic data, which limit scalability and introduce potential sim-to-real gaps. Moreover, although these methods have improved spatiotemporal understanding, they still lack strong metric reasoning capabilities for dynamic scenes, such as estimating object motion in real-world units. Prior work has explored LiDAR-based metric depth supervision to enhance spatial perception, but it does not directly address temporal reasoning. We introduce STAR-VLM, an automotive radar-supervised framework that enhances spatiotemporal VLMs with motion reasoning and metric velocity estimation for autonomous driving. Automotive radar is a low-cost and widely deployed sensor that provides complementary spatiotemporal supervision through range and Doppler measurements. By leveraging these measurements as label-free ground truth during training, STAR-VLM improves the metric spatiotemporal reasoning ability of VLMs. Through experiments on driving scenarios, we show that STAR-VLM achieves state-of-the-art performance on both motion classification and metric velocity estimation, outperforming even task-specific methods designed for each task. These results highlight automotive radar as a scalable and cost-effective source of supervision for building metric-aware spatiotemporal VLMs for real-world autonomous driving.