Hybrid CPU-GPU framework cuts VLA inference latency by up to 30.9%

Deploying Vision-Language-Action (VLA) models in autonomous vehicles is notoriously hard: they demand high GPU memory and low-latency inference, while legacy vehicle platforms were designed for modular pipelines that underutilize CPU resources. A new arXiv paper (2608.14586) from Haibo Hu and colleagues introduces a hybrid CPU-GPU inference framework that redistributes compute across heterogeneous processors. The design partitions the VLA backbone at block-layer granularity, executing the visual encoder and large language model (LLM) prefix on the GPU while offloading the LLM suffix to the CPU through a cross-frame asynchronous pipeline. This exposes a schedulable boundary that lets the system balance memory and compute pressure across the vehicle's processors.

The framework was evaluated on two representative driving VLA models, Orion and MindDrive, using the Bench2Drive benchmark. Results show average latency reduced from 521ms to 408ms for Orion (a 21.7% improvement) and from 443ms to 306.2ms for MindDrive (30.9% improvement). For Orion, estimated peak GPU memory dropped from 45GB to 29GB. In a real-vehicle deployment test alongside the full autonomous driving stack, native Orion could not run within the onboard GPU memory budget, but the hybrid version ran successfully. This approach directly addresses the GPU resource bottleneck, making unified VLA models practical on existing hardware without requiring a full platform redesign. The work suggests that hybrid CPU-GPU execution is a viable path for bringing advanced AI models into production vehicles.