Talk2Car: Taking Control of Your Self-Driving Car

The evolution of autonomous driving has transitioned from static object detection to dynamic scene understanding, integrating multi-modal sensors like cameras, LiDAR, and radar. Early datasets such as MSCOCO and Cityscapes provided static images for object detection, but lacked real-world complexity. The ReferIt and RefCOCO series introduced language grounding but mainly focused on controlled environments. Recent efforts like Cityscapes-Ref and CLEVR-Ref extended to urban scenes and synthetic data, respectively. However, these datasets often lack the temporal and multi-sensor richness of real city driving scenarios. The nuScenes dataset, with its multi-view videos, LiDAR, and 3D annotations, offers a comprehensive platform. Building upon this, Talk2Car introduces natural language commands in realistic urban contexts, emphasizing the need for models to understand spatial relations, attributes, and long sentences in dynamic environments.

The core challenge is enabling autonomous vehicles to interpret natural language commands accurately within complex urban scenes. This involves understanding spatial relationships, attributes, and contextual cues embedded in free-form language, and mapping them onto visual targets. Existing methods struggle with ambiguity, long sentences, and distant objects, limiting real-world applicability. Achieving high accuracy and low latency is crucial for safety and user experience. The problem is compounded by the diversity of urban environments, sensor noise, and linguistic vagueness. Addressing these issues requires datasets that reflect real-world complexity and models capable of multi-source reasoning under computational constraints.

The paper's key innovations include: 1) the creation of Talk2Car, a large-scale, multimodal dataset with natural language commands grounded in urban scenes, incorporating LiDAR and multi-view videos; 2) the integration of spatial relationship encoding and attribute-based reasoning into deep models; 3) comprehensive evaluation of state-of-the-art models like MAC and Stack-NMN in realistic settings, establishing new benchmarks. The dataset's diversity and realism enable training models that better generalize to real-world scenarios. The model architecture emphasizes multi-step reasoning, multi-source fusion, and spatial understanding, addressing limitations of prior static or synthetic datasets. This approach pushes the frontier of natural language grounding in autonomous driving.

• �� Data collection: Using nuScenes videos, annotators generate natural language commands referring to objects with spatial and attribute details, linked to 3D bounding boxes. • Feature extraction: Visual features are extracted via ResNet-101; LiDAR data and spatial relations are encoded as additional features. • Candidate region generation: SSD-512-based RPN produces 64 interest regions per image. • Language encoding: Bi-LSTM encodes commands, capturing long-range dependencies. • Model architecture: • MAC and Stack-NMN models perform multi-step reasoning, attending to language parts and visual features iteratively. • Spatial encoding: Coordinates and spatial relations are embedded to improve disambiguation. • Training: Loss functions combine classification and bounding box regression, optimized with Adam. • Evaluation: IoU threshold 0.5 used to determine correct predictions, with metrics for speed and parameter count.

The dataset is split into 70% training, 10% validation, 20% testing, ensuring city and scene diversity. Multiple models are trained and evaluated, including simple baselines (random, heuristic) and advanced deep models (MAC, Stack-NMN). Performance metrics include IoU, inference time, and parameter count. Ablation studies test the impact of spatial encoding and multi-modal fusion. The models are challenged with varying command complexity, object distance, and scene clutter. Results demonstrate that MAC outperforms others, with significant improvements over baselines, especially in complex spatial and attribute-based commands. Cross-scenario robustness and real-time feasibility are also analyzed.

MAC achieves 50.51% IoU at 0.5 threshold, surpassing Stack-NMN's 33.71%. It maintains robustness across complex, long, and distant commands, with inference speed around 51ms. Ablation shows multi-modal fusion and spatial encoding are critical for performance. The results highlight the importance of deep reasoning and multi-source features, especially in urban, cluttered scenes. The models handle ambiguous and multi-object scenarios better than simple baselines, but still face challenges with extreme complexity and real-time deployment.

This technology enables natural language interfaces for autonomous vehicles, improving passenger interaction and safety. It can be integrated into in-car systems for navigation, parking, and dynamic scene management. The approach supports multi-object and multi-relationship understanding, essential for complex urban environments. Long-term, it could facilitate fully autonomous, intuitive, and safe transportation, reducing driver workload and enhancing user experience. Deployment requires optimization for low-power hardware and robustness to linguistic ambiguity.

Models struggle with highly complex, multi-object commands involving multiple relations. Dataset bias towards specific cities limits generalization. High computational cost and large model size hinder deployment on embedded systems. Ambiguity and vagueness in natural language still cause errors, necessitating advanced NLP techniques. Future work should focus on dataset expansion, model simplification, and better handling of linguistic ambiguity to improve robustness and efficiency.