MILD: Mediator Agent System with Bidirectional Perception and Multi-Layered Alignment for Human-Vehicle Collaboration

The evolution of autonomous driving has transitioned from manual control to semi-automated and fully autonomous systems, exemplified by SAE levels 2-5. Early systems like Tesla Autopilot and Waymo focused on environment perception and control. Recent advances leverage deep multimodal perception (LiDAR, cameras) and large language models (e.g., GPT series) for reasoning and interaction. Despite progress, most systems operate unidirectionally, lacking dynamic understanding of driver states and mutual communication. This results in reduced trust, safety concerns, and regulatory challenges. The need for transparent, aligned, and auditable systems has become urgent, especially as vehicles become more autonomous and human drivers transition to active managers rather than passive supervisors.

Current systems often fail to effectively perceive and interpret driver internal states and external traffic context simultaneously, leading to misalignment and safety risks. The weak bidirectional communication hampers shared situational awareness, especially in complex or unpredictable scenarios. Existing approaches lack mechanisms for enforcing strict safety constraints and providing explainable, auditable strategies. This gap limits the deployment of trustworthy autonomous vehicles, as stakeholders demand systems that are not only safe but also transparent and compliant with regulations. Overcoming these challenges requires integrating multimodal perception, structured policy schemas, and verification mechanisms into a unified framework.

The paper introduces several key innovations: 1) a joint perception model that fuses in-cabin and out-of-cabin multimodal data into structured summaries, enabling comprehensive understanding; 2) a schema-driven policy language that formalizes high-level strategies, ensuring interpretability and auditability; 3) ECPO, an optimization framework that incorporates evidence grounding and multi-source constraints via automatic validators, providing verifiable compliance; 4) a retrieval-augmented mechanism that dynamically injects traffic laws, vehicle limits, and driver preferences into decision-making. These innovations collectively enable a transparent, safe, and adaptable human-vehicle collaboration system, addressing core limitations of prior work.

• �� The perception agent receives multi-view sensor data (cameras, radar, driver monitoring) and encodes this into structured labels and scene summaries, capturing driver states (distraction, gaze) and environmental context (traffic density, maneuvers). • The strategy agent, based on a lightweight GPT-3.5 model, takes the structured perception output, driver and vehicle profiles, and retrieved constraints (traffic laws, preferences) to generate high-level, structured policies following the PolicyAction schema. • Policies specify objectives, constraints, and atomic actions (e.g., driving advice, HMI prompts), with rationales referencing perception evidence and retrieved snippets. • ECPO employs a validator that assesses each policy against multi-source constraints, scoring based on safety violations, evidence coverage, and structural completeness. • During training, a teacher-student pipeline refines policies from GPT-3.5 outputs, with manual auditing ensuring quality. • Offline evaluation compares perception accuracy, policy compliance, and human ratings across datasets, validating the system’s effectiveness.

The evaluation uses BDD100K, nuScenes, and Drive360 datasets, comparing MILD with baseline models like rule-based systems and pure LLM approaches. Metrics include perception accuracy, policy compliance rate, violation count, and human ratings on adequacy, comfort, and explainability. Hyperparameters involve a 7-billion-parameter perception model, GPT-3.5 for policy generation, and ECPO weights tuned between 0.1-0.3. Ablation studies isolate the impact of perception, retrieval, and validation components. Results show a 12% increase in perception accuracy, 15% in policy compliance, and a 20% reduction in violations, with human scores exceeding 4.5/5 for policy adequacy and interpretability.

MILD consistently outperforms baselines in perception and policy metrics. The perception module achieves 12% higher accuracy, while the policies generated meet safety constraints 15% more often. Human evaluations rate the policies at 4.5/5, citing better adaptability and explanation. The ECPO optimizer effectively enforces constraints, reducing violations by 20%. These results demonstrate the system’s robustness across diverse traffic scenarios and its potential for real-world deployment.

This system can be integrated into advanced driver assistance systems (ADAS) and autonomous vehicles to improve safety, transparency, and user trust. It is particularly suited for complex urban environments, where dynamic perception and strict regulatory compliance are critical. The framework also supports personalized driving assistance, enhancing user experience and acceptance. Long-term, it can serve as a foundation for fully autonomous, human-centric transportation networks, enabling safer and more explainable mobility solutions.

Despite promising results, the system’s performance in highly unpredictable or rare scenarios remains limited, mainly due to data coverage gaps. The reliance on predefined schemas and retrieval mechanisms may restrict flexibility in novel situations. Computational demands for real-time inference are still high, requiring further optimization. Future work should focus on end-to-end learning, reinforcement-based adaptation, and expanding the system’s ability to handle edge cases and extreme events.