Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportation
Sky-Drive combines CARLA, RPC, digital twins, and bidirectional human-AI mentoring; the paper reports no quantitative performance gains.
Key Findings
Methodology
Sky-Drive uses CARLA as its core engine and extends it with rpclib-based remote procedure calls for synchronized, cross-terminal simulation. Its human-in-the-loop layer combines steering wheels, virtual reality, cameras, smartwatches, and eye tracking to capture control and behavioral signals. HAIM, or Human as AI Mentor, injects human preferences and transportation knowledge into learning, while AIHM, or AI as Human Mentor, delivers guidance and personalized training. A digital-twin pipeline combines HD maps, roadside sensors, traffic cameras, historical records, and lab-vehicle data.
Key Results
- The paper demonstrates an end-to-end workflow spanning scenario generation, data collection, algorithm training, and hardware validation, but reports no accuracy, collision-rate, latency, or baseline-improvement numbers. Implemented capabilities include CARLA customization, synchronized AV/HV/pedestrian simulation, and Socket.IO-based online monitoring.
- Table I marks Sky-Drive as supporting all six listed capabilities: distributed multi-agent simulation, digital-twin environments, traffic-flow modeling, AI frameworks, human-in-the-loop interfaces, and hardware integration. Most comparison systems, including CARLA, SUMO, MetaDrive, and Waymax, support only subsets of this combination.
- The personalized-driving case converts feedback such as “It is too fast” or “the acceleration is too harsh” into inferred comfort preferences, with LLM processing proposed for this step. However, the case is a workflow demonstration rather than a controlled learning study and gives no sample count, training iterations, or generalization metrics.
Significance
The work shifts driving simulation from single-machine, one-way policy testing toward distributed, bidirectional human-AI interaction. It targets persistent barriers: dangerous collection of rare interactions, limited access to authentic human behavior, and the absence of realistic environments for socially-aware driving. Academically, the platform can support intention modeling, human-in-the-loop reinforcement learning, and mixed-traffic studies. Industrially, its digital-twin, ROS, and remote-operation interfaces could connect simulation, proving grounds, and physical vehicles.
Technical Contribution
The principal contribution is systems integration: CARLA sensing and visualization are combined with RPC-based distributed control, Socket.IO monitoring, multimodal human sensing, digital twinning, and planned hardware-in-the-loop testing. HAIM and AIHM define a closed exchange rather than treating humans only as low-level controllers. The design also exposes transportation-science priors such as IDM and MOBIL for socially-aware policies. The paper does not introduce a new reinforcement-learning objective, formal optimality guarantee, or quantitative algorithmic theorem.
Novelty
The novelty lies mainly in the integrated research loop, not in a new standalone learning algorithm. Relative to CARLA, SUMO, MetaDrive, SMARTS, and Waymax, Sky-Drive emphasizes real-time, cross-computer participation, richer human sensing, and reciprocal adaptation. Relative to human-in-the-loop tools, it connects interaction data to digital twins, social traffic models, remote physical vehicles, and future foundation-model support. Claims of being the first should remain limited to the demonstrated system combination because the paper lacks broad empirical comparison.
Limitations
- The manuscript is primarily an architecture and workflow paper. It provides no public benchmark, statistical significance analysis, or measurements of synchronization error, throughput, communication latency, or policy performance against CARLA, SUMO, MetaDrive, or other baselines.
- LLM/VLM integration and the complete hardware-in-the-loop framework are planned rather than fully evaluated. Details on preference inference, multimodal time alignment, network jitter, remote takeover safety, privacy, and failure containment remain limited.
- Digital-twin fidelity depends on HD maps, sensor coverage, and data quality. The paper does not systematically test map error, missing modalities, domain shift, privacy constraints, or transfer across locations.
Future Work
The authors plan system- and agent-level LLM/VLM integration for global feedback, contextual understanding, natural communication, and personalized decision-making. They also plan ROS-based hardware-in-the-loop validation with physical vehicles. Strong follow-up work should release standardized multimodal datasets, report network and synchronization metrics, define socially-aware evaluation protocols, and compare against CARLA, SMARTS, Waymax, Nocturne, and real driving data under reproducible conditions.
AI Executive Summary
Autonomous-driving research is moving beyond the question of whether a vehicle can drive alone. Future traffic will mix autonomous vehicles, human drivers, pedestrians, robots, and intelligent infrastructure, all negotiating space continuously. Platforms such as CARLA, SUMO, MetaDrive, and Waymax have made controlled testing practical, but most operate on one machine, treat people as low-level input sources, or optimize vehicle-level metrics. Real-road data collection is also expensive, dangerous, and poorly suited to rare interactions.
Sky-Drive proposes a broader research environment. Built on CARLA and Unreal Engine, it uses rpclib-based RPC to synchronize agents across terminals and Socket.IO for monitoring. Steering wheels, VR systems, cameras, smartwatches, and eye trackers record human behavior. The HAIM module lets people provide preferences, situational knowledge, and norms; AIHM returns guidance and personalized training. Its digital-twin layer combines HD maps from lab vehicles, open data, traffic cameras, roadside sensing, and historical records. Planned LLM/VLM and ROS hardware-in-the-loop components would extend the loop into natural-language interaction and physical validation.
The evidence is architectural rather than quantitative. Table I indicates that Sky-Drive supports six capability categories simultaneously: distributed multi-agent simulation, digital twins, traffic-flow modeling, AI frameworks, human-in-the-loop interfaces, and hardware integration. A personalized-driving demonstration shows how comments such as “too fast” could inform comfort modeling. Yet the paper reports no collision reduction, latency, sample size, benchmark score, or statistical comparison. Its central promise is therefore a unified infrastructure for future experiments, not a proven superiority claim.
Deep Analysis
Background
Driving simulators now cover several niches. CARLA and LGSVL provide comprehensive environments; SUMO, Vissim, and CityFlow model traffic flow; AirSim targets sensor simulation; Highway-Env, SMARTS, and MetaDrive support policy learning; CarSim and IPG CarMaker emphasize vehicle dynamics; Waymax, ScenarioNet, and Nocturne use data-driven scenarios. These systems have reduced testing cost, but most do not support real-time human participation across computers, rich behavioral sensing, or continuous bidirectional knowledge exchange.
Core Problem
Future transportation is a mixed ecosystem of AVs, human-driven vehicles, pedestrians, delivery robots, and infrastructure agents. Success requires not only safety, efficiency, and route completion, but also intention recognition, comfort adaptation, implicit right-of-way negotiation, and collective traffic performance. The problem is difficult because authentic interaction data are risky and sparse, human feedback is often reduced to control signals, social objectives are not captured by single-vehicle metrics, and simulation must remain connected to physical vehicles and real environments.
Innovation
- �� Distributed architecture: RPC lets separate terminals control separate agents while sharing state. • Multimodal human-in-the-loop: steering wheels, VR, cameras, smartwatches, and eye tracking capture control, speech, gaze, and behavior. • HAIM/AIHM: humans provide preferences and norms; AI provides advice and training. • Digital twin: HD maps, open data, traffic cameras, roadside sensors, and vehicle data create virtual replicas. • Planned extensions: LLM/VLM reasoning and ROS-based hardware-in-the-loop validation.
Methodology
- �� Scene generation: CARLA and Unreal Engine create configurable road networks, traffic rules, weather, and agents; digital-twin processing maps categorized real-world data into virtual scenes. • Distributed simulation: host and client terminals communicate over LAN/VLAN; RPC synchronizes AV, HV, pedestrian, vehicle, and sensor states; Socket.IO enables monitoring. • Human sensing: synchronized streams combine control inputs, gaze, video, voice, and behavioral signals. • Collaborative learning: HAIM incorporates feedback and IDM/MOBIL transportation priors; AIHM produces real-time guidance. • Reality validation: trained policies connect through ROS to a lab-developed Ford E-Transit electric van, while a 5G mmWave hotspot supports remote monitoring and takeover.
Experiments
This is a platform demonstration rather than a conventional machine-learning benchmark. Scenarios come from CARLA/Unreal Engine, HD maps collected by lab-developed vehicles, open-source data, and real traffic data. Physical testing is associated with the Madison College Public Safety Training Center and MGA Research Corporation. Comparisons include Nvidia Drive Sim, CARLA, SUMO, MetaDrive, SMARTS, Waymax, Nocturne, and others. The manuscript gives no dataset sizes, training hyperparameters, statistical tests, ablations, or quantitative latency and performance metrics.
Results
The clearest result is capability coverage: Table I marks Sky-Drive as supporting all six categories, whereas representative platforms usually lack distributed multi-terminal operation, digital twins, or hardware interfaces. The personalized-driving example shows how natural feedback can be converted into comfort information. Because no collision rate, synchronization error, communication latency, accuracy, or baseline comparison is reported, the evidence establishes architectural feasibility, not superiority of a learned driving policy.
Applications
Sky-Drive can support AV-HV-pedestrian interaction modeling, safe collection of rare scenarios, human-in-the-loop reinforcement learning, socially-aware policy training, personalized speed and comfort modeling, customized scenario generation, and ROS-connected vehicle validation. Practical deployment requires synchronized sensors, reliable networking, privacy safeguards, drive-by-wire interfaces, controlled testing sites, and robust emergency takeover procedures.
Limitations & Outlook
The current evidence is largely descriptive. Scalability, packet loss, timing drift, RPC latency, sensor synchronization, and large-agent performance are not measured. LLM/VLM and full hardware-in-the-loop integration remain future work and may introduce hallucinations, unpredictable recommendations, and new cybersecurity risks. Future studies should release multimodal datasets, define social metrics covering risk, flow, comfort, and courtesy, conduct ablations of HAIM and AIHM, and compare against CARLA, SMARTS, Waymax, Nocturne, and real-world driving data across locations.
Plain Language Accessible to non-experts
Imagine a large driving school spread across several rooms. In an ordinary simulator, one student drives on one computer while computer-controlled cars fill the road. Sky-Drive connects the rooms so that one person can drive a car, another can play a pedestrian, and a third can control another vehicle, while everyone sees the same traffic situation in real time.
The school also uses many kinds of records. A steering wheel shows how someone controls the car, cameras show what they are doing, eye tracking shows where they look, and a smartwatch can capture additional behavior. This is more informative than simply recording whether the car stayed in its lane: it can reveal hesitation, confidence, comfort, and the reason for a decision.
There are two kinds of coaches. One lets people teach the computer: “That speed feels too high.” The other lets the computer coach people with warnings and practice. The system can also copy a real road into a computer version using maps, cameras, sensors, and vehicle data. The important caveat is that the paper mainly describes how this school is built. It does not yet publish exam scores showing that its students outperform other schools.
ELI14 Explained like you're 14
Think of Sky-Drive as a multiplayer racing game where not everyone is controlled by the computer. You might drive an autonomous car, your friend might act as a pedestrian, and another player might drive a normal car. They can even use different computers or be in different places, but the road and traffic still have to match perfectly. That lets researchers study the awkward moments real drivers deal with, like who should yield at an intersection.
The game watches more than crashes. It can record steering, cameras, eye movements, smartwatch signals, and comments such as “Whoa, that is too fast!” The computer may learn that a particular driver prefers gentle acceleration and a slower cornering style. Why force everyone to use one personality when real drivers are different?
Sky-Drive has two coaching directions. Humans teach the AI what feels safe, polite, or comfortable, while the AI teaches humans through suggestions and practice. It can also rebuild a real place inside the computer using maps and traffic data, so dangerous situations can be tested without putting anyone on an actual road.
The exciting part is the planned future: language-and-vision AI could understand natural instructions, and real cars could test the computer’s decisions in controlled places. But this is not a game leaderboard yet! The paper does not say how many crashes were prevented or how fast the system is. Network delays, privacy, AI mistakes, and emergency control still need serious testing.
Glossary
Distributed Multi-Agent Simulation
A simulation in which independent agents run on separate computers while sharing a synchronized world state. It is different from running many agents on one machine.
Sky-Drive uses CARLA and rpclib-based RPC to coordinate AVs, human-driven vehicles, and pedestrians across terminals.
Human-in-the-Loop
A setup in which real people participate in system operation, data collection, or learning rather than merely supplying offline labels. Their actions and feedback affect the live simulation.
The platform combines steering wheels, VR, cameras, smartwatches, and eye tracking.
Digital Twin
A virtual representation of a real environment that is constructed and updated from physical-world data. Its value depends on the fidelity and freshness of maps, sensors, and traffic information.
Sky-Drive integrates HD maps, traffic cameras, roadside sensors, open data, and lab-vehicle measurements.
HAIM / AIHM
HAIM means Human as AI Mentor: people provide preferences, experience, and norms to AI. AIHM means AI as Human Mentor: AI provides guidance, feedback, and personalized training.
Together they define the platform’s bidirectional human-AI knowledge exchange.
IDM / MOBIL
The Intelligent Driver Model describes longitudinal following behavior; MOBIL models lane changes using incentives and their effects on braking. Both are established transportation-science models.
The paper identifies them as interpretable priors for socially-aware autonomous driving.
Hardware-in-the-Loop
A validation method that connects simulation algorithms to real controllers or vehicles under controlled conditions. It provides an intermediate step between pure simulation and open-road deployment.
Sky-Drive plans ROS integration with a Ford E-Transit and 5G-supported remote operation.
Open Questions Unanswered questions from this research
- 1 Can the platform preserve synchronization under latency, packet loss, and many agents? Public measurements of end-to-end delay, timing drift, throughput, and scalability are needed.
- 2 How reliably can language, gaze, and control signals be converted into stable preferences? The paper lacks multimodal datasets, annotation protocols, and comparisons with human or rule-based preference models.
- 3 What should socially-aware driving optimize? Future benchmarks must jointly measure collision risk, traffic flow, collective comfort, courtesy, intention understanding, and individual preference satisfaction.
Applications
Immediate Applications
Rare-interaction data collection
Universities and research teams can use multiple terminals to reproduce near-crashes, yielding, and informal right-of-way negotiations among AVs, human drivers, and pedestrians. Requirements include CARLA, synchronized networking, participant interfaces, and sensor logging. The output can support behavior modeling and policy evaluation.
Personalized driving training
Vehicle companies can combine speed, acceleration, gaze, and verbal feedback to estimate individual comfort preferences through HAIM and AIHM. The resulting assistant can provide personalized advice in virtual roads and later be checked through ROS-connected vehicles at controlled test sites.
Long-term Vision
City-scale digital-twin laboratory
Transportation agencies could combine HD maps, roadside sensing, and traffic cameras to evaluate AV penetration, signal strategies, and mixed-traffic designs in a virtual city. Progress requires common data standards, privacy protection, continuous updating, and interoperability across institutions.
Human-centered autonomy validation chain
With LLM/VLM reasoning, 5G teleoperation, and ROS hardware-in-the-loop, the platform could connect natural-language requirements, simulation training, and physical-vehicle validation. Major obstacles include model reliability, cybersecurity, liability, certification, and safe fallback behavior.
Abstract
Recent advances in autonomous system simulation platforms have significantly enhanced the safe and scalable testing of driving policies. However, existing simulators do not yet fully meet the needs of future transportation research-particularly in enabling effective human-AI collaboration and modeling socially-aware driving agents. This paper introduces Sky-Drive, a novel distributed multi-agent simulation platform that addresses these limitations through four key innovations: (a) a distributed architecture for synchronized simulation across multiple terminals; (b) a multi-modal human-in-the-loop framework integrating diverse sensors to collect rich behavioral data; (c) a human-AI collaboration mechanism supporting continuous and adaptive knowledge exchange; and (d) a digital twin framework for constructing high-fidelity virtual replicas of real-world transportation environments. Sky-Drive supports diverse applications such as autonomous vehicle-human road users interaction modeling, human-in-the-loop training, socially-aware reinforcement learning, personalized driving development, and customized scenario generation. Future extensions will incorporate foundation models for context-aware decision support and hardware-in-the-loop testing for real-world validation. By bridging scenario generation, data collection, algorithm training, and hardware integration, Sky-Drive has the potential to become a foundational platform for the next generation of human-centered and socially-aware autonomous transportation systems research. The demo video and code are available at:https://sky-lab-uw.github.io/Sky-Drive-website/