Efficient Block-Layer Parallel Inference for Vision-Language-Action on Hybrid Architectures

Recent advances in autonomous driving have shifted from modular pipelines toward end-to-end models that unify perception, reasoning, and control. Large multimodal models like LMDrive, CoVLA, and AutoVLA exemplify this trend, leveraging transformer architectures for scene understanding and decision-making. Industrial implementations such as Xiaomi's ORION and MindDrive demonstrate practical deployment efforts. However, these models' computational demands—especially in perception and reasoning—pose significant challenges for real-time inference on embedded vehicle hardware. Existing solutions focus on model compression, layer skipping, or quantization, but often neglect system-level resource management and heterogeneous hardware utilization, which are critical for scalable deployment.

The core challenge lies in executing large VLA models efficiently within the limited GPU memory and computational budget of autonomous vehicles. High latency and memory bottlenecks hinder real-time responsiveness, essential for safe driving. Traditional serial inference, where the entire model runs on GPU, leads to underutilized CPU resources and excessive memory consumption. Balancing the workload between CPU and GPU while maintaining model accuracy and low latency remains unresolved, especially under continuous multi-frame streaming scenarios with tight real-time constraints.

The paper introduces a block-layer partitioning strategy that exploits the hierarchical structure of transformer models, enabling natural division at Transformer blocks. This preserves model integrity and facilitates heterogeneous execution. Additionally, a cross-frame asynchronous pipeline overlaps GPU prefix computation with CPU suffix processing across consecutive frames, significantly reducing end-to-end latency. A dynamic resource scheduler adjusts the layer split point based on runtime conditions, balancing load and communication overhead. Hardware-aware acceleration techniques further optimize CPU inference. These innovations collectively enable efficient, scalable deployment of VLA models in resource-constrained autonomous vehicles.

• �� Model partitioning at Transformer block layer, dividing the backbone into GPU prefix and CPU suffix, ensuring structural integrity.
• �� Implementation of a cross-frame asynchronous pipeline, where GPU processes current frame's prefix while CPU handles previous frame's suffix, overlapping computation.
• �� Dynamic adjustment of partition point via resource-aware scheduler, based on real-time system load and memory availability.
• �� Hardware-aware CPU acceleration, utilizing optimized libraries to enhance inference speed.
• �� Validation on Orion and MindDrive models, measuring latency, memory, and inference frequency.
• �� Deployment on real vehicle platform with Autoware.Universe, demonstrating practical feasibility.
• �� Ablation studies to analyze the impact of partition point, pipeline overlap, and scheduling strategies.

The evaluation employed Bench2Drive benchmarks, comparing the proposed hybrid framework against baseline full-GPU inference. Metrics included average latency, GPU peak memory, and inference frequency. Various layer split points were tested to identify optimal trade-offs. Real vehicle tests involved integrating the system with Autoware.Universe, assessing stability, responsiveness, and resource utilization under actual driving conditions. Hyperparameters such as batch size, frame rate, and communication buffer sizes were tuned for optimal performance. Ablation experiments isolated the effects of pipeline overlap and dynamic scheduling, confirming their contributions to efficiency gains.

Results show that the hybrid approach reduces Orion's average latency from 521ms to 408ms, a 21.7% improvement, and MindDrive from 443ms to 306.2ms, a 30.9% reduction. GPU peak memory consumption drops from 45GB to 29GB, alleviating hardware constraints. The cross-frame asynchronous pipeline maintains high inference frequency (>3Hz), suitable for real-time control. In real vehicle deployment, the hybrid system successfully integrated with the full vehicle stack, whereas native Orion failed due to memory limitations. These findings demonstrate the method's effectiveness in balancing latency, memory, and system integration challenges.

This framework is directly applicable to autonomous vehicles requiring real-time perception and decision-making with constrained hardware resources. It enables scalable deployment of large multimodal models, improving responsiveness and safety. The approach can be extended to multi-task learning, multi-modal fusion, and adaptive resource management, supporting future intelligent transportation systems. Its flexibility allows integration with existing perception stacks, facilitating incremental upgrades and system robustness.

The approach relies on Transformer block structures, limiting applicability to models with different architectures. Dynamic scheduling may face challenges under highly variable system loads, affecting stability. Communication overhead, although minimized, could still impact ultra-low latency scenarios. Further research is needed to generalize the method across diverse hardware platforms and model types, and to optimize scheduling under extreme conditions.