Evolving Cache Schedules for Fast Diffusion Policy Inference

Diffusion policies have gained attention in robotic control for their ability to model multimodal action distributions through conditional denoising processes. Despite their effectiveness in complex manipulation settings, the iterative denoising loop imposes a heavy inference burden, limiting real-time control capabilities. Existing acceleration methods often rely on reducing denoising steps or pruning and distillation to reduce computation, but these typically require additional training or alter the sampling process.

The core problem with diffusion policies is the high computational cost of the iterative denoising process, which directly limits achievable action frequency, making it difficult to meet the low-latency requirements of real-time, smooth robotic control. Existing cache methods fail to leverage inter-block redundancy effectively, leading to suboptimal performance-efficiency trade-offs.

EVO's core innovation lies in its global cache scheduling strategy, optimizing cache refresh schedules via evolutionary search to skip redundant computation blocks. EVO introduces redundancy-aware initialization and target-conditioned early stopping to make the search practical. Compared to existing methods, EVO significantly improves the performance-efficiency trade-off.

• �� EVO uses evolutionary search to optimize cache refresh schedules, representing each candidate as a complete schedule over the block-timestep lattice.
• �� Redundancy-aware initialization biases the initial population toward less redundant block-timestep positions using activation redundancy.
• �� Target-conditioned early stopping terminates the search once a satisfactory schedule is found, reducing unnecessary evaluations.
• �� The selected cache schedule can be directly applied to pretrained diffusion policies without retraining.

Experiments were conducted on multiple robotic manipulation benchmarks, including Push-T, Block Push, and Kitchen tasks. The pretrained DP-T policy was used as the baseline to evaluate EVO's performance against representative training-free caching methods. Success rate was used as the primary performance metric, and the FLOPs required for action generation and relative speedup were reported.

EVO preserves near-full performance on multiple manipulation benchmarks while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G. In the Push-T task, EVO significantly improved inference speed by skipping redundant computation blocks while maintaining closed-loop performance.

EVO can be directly applied to real-time robotic control tasks in visuomotor control, particularly in scenarios requiring high action frequency and low latency. Its training-free nature makes it easy to integrate into existing systems.

EVO's performance depends on the accuracy of the initial redundancy-aware initialization, which may lead to suboptimal results in some cases. The computational cost of evolutionary search is high, potentially limiting its applicability in all real-time scenarios. Future work could explore more efficient search algorithms to further reduce computational costs.