ARTEMIS: Autoregressive End-to-End Trajectory Planning with Mixture of Experts for Autonomous Driving

Autonomous driving has largely evolved along two lines. Modular stacks separate perception, prediction, and planning, which improves interpretability but also accumulates interface errors. End-to-end methods remove those boundaries and map raw sensors directly to trajectories or controls, but many remain static, one-shot predictors. Recent works such as Transfuser, UniAD, VADv2, Hydra-MDP, and DiffusionDrive improved multimodal fusion, unified learning, and generative planning, yet the temporal evolution of driving decisions remains hard to capture. ARTEMIS builds on this landscape by combining autoregression with expert specialization.

The core problem is how to plan trajectories in an end-to-end system when three difficulties coincide: temporal dependence, multimodal behavior, and scene diversity. One-shot prediction tends to flatten the relationship among future waypoints, while a single network can average together incompatible behaviors such as straight driving, turning, merging, and roundabout navigation. The paper also observes strong command imbalance in navtrain and some mismatches between control commands and expert trajectories, meaning explicit command-based routing can mislead expert learning and degrade trajectory quality.

ARTEMIS makes three notable innovations. First, it turns planning into autoregressive generation, so each waypoint is predicted from the ego state, time embedding, and previous waypoints, preserving causal structure. Second, it introduces MoE into end-to-end driving, using shared and private experts to represent both general and scenario-specific behaviors. Third, it adds a lightweight batch reallocation strategy that rearranges samples by expert activation, reducing unnecessary computation during training. A final refinement module then enforces semantic and kinematic plausibility, pushing the output beyond raw waypoint sampling.

• �� Perception: 8 camera views and LiDAR are encoded with a Transfuser-like dual-stream design; a ResNet-34 backbone extracts features and fuses them into BEV representation F_bev.
• �� State encoding: only the current ego state s0 is encoded, including control commands, 2D velocity, and acceleration, to avoid causal confusion from directly learning from historical ego trajectories.
• �� Temporal and positional embeddings: time embedding T E_t marks the target step; positional embedding P E_t is added only in the initial autoregressive step to avoid repeated noise accumulation.
• �� Autoregressive generation: the concatenated query C_t = Concat(T E_t, Q_s, Q(1:t)) is passed through a Transformer encoder with a padding mask so the current step attends only to historical planning queries.
• �� MoE routing: a two-MLP router computes expert scores; top-k=2 experts are activated. The batch is then sorted by expert indices, and contiguous blocks with identical indices are identified.
• �� Batch reallocation: F_bev and the planning queries are rearranged by π_i; blocks {E^i_j, n^i_j} are processed by E_shared and E_private, then restored to the original order and fused with weights g_ij.
• �� Probabilistic waypoint sampling: an MLP predicts mean and variance for position and heading, and each waypoint is sampled as y_t ~ N(μ_t, σ_t^2).
• �� Refinement: semantic kinematic optimization encodes a BEV semantic map with a CNN, encodes the initial trajectory with a GRU, and optimizes point by point; cross-attention refinement then fuses agent queries Q_agent and ego queries Q_ego.
• �� Training: a staged schedule first trains perception auxiliaries, then end-to-end planning; losses are L_perception=λ_semL_sem+λ_classL_class+λ_boxL_box and L_planning=λ_trajL_traj+λ_NLLL_NLL+L_perception.

Experiments are conducted on NAVSIM, a benchmark curated from OpenScene with difficult scenarios only. The training split has 1,192 scenarios and the test split has 136. Each sample includes 8 camera views, LiDAR fused from 5 sensors, map annotations, and 3D bounding boxes. The model uses 4 frames of 2 seconds historical/current context to predict a 4-second trajectory with 8 future steps. Implementation details include Transfuser perception, ResNet-34 backbone, 5 private experts, 1 shared expert, top-k=2 routing, batch size 128 on 2 A100 GPUs, learning rate 2e-4, and weight decay 1e-4. Loss weights are set separately for perception pretraining and end-to-end training.

The main quantitative result is 87.0 PDMS on navtest, with strong component scores such as NC 98.3, DAC 95.1, EP 81.4, TTC 94.3, and C 100. Compared with Transfuser’s 84.0 and Hydra-MDP++’s 86.6, this is a clear improvement within the same general end-to-end family. On the extended benchmark, ARTEMIS scores 83.1 EPDMS and surpasses Hydra-MDP++’s 80.6, especially on TTC 97.4, LK 96.5, and EC 98.3. A careful reading shows that DiffusionDrive reports 88.1 PDMS, so the strongest claim is not absolute PDMS dominance but superior extended-metric performance and strong behavior under diverse scenarios.

ARTEMIS is well suited to urban autonomous driving where a planner must handle intersections, roundabouts, merges, and lane selection under ambiguous cues. Because it outputs waypoints sequentially, it can naturally support online decision making and incremental correction. In research, it can serve as a benchmark for comparing autoregressive planning, diffusion planning, and MoE-based specialization. In industry, it suggests a route toward more scalable planning stacks that distribute different maneuvers to different experts, though deployment will require further latency and safety validation.

The method depends on the NAVSIM evaluation protocol and does not yet prove broad robustness beyond that dataset. MoE adds routing and reorganization complexity, and the paper does not fully report end-to-end inference latency or on-vehicle cost. The decision to omit expert balance loss may help preserve specialization, but it can also increase routing imbalance, which future work should quantify more thoroughly.