PyTorch adds oneDNN LSTM primitive for XPU, boosting inference 6x
New optimization cuts LSTM latency from 34ms to 5ms on Intel XPU
This PyTorch pull request (viable/strict/1780997283) adds onednn LSTM primitive support for XPU inference, leveraging Intel's oneAPI Deep Neural Network Library (oneDNN). The core change replaces the existing per-timestep fused cell approach—which sequentially iterates over each time step with individual kernel launches—with a single `dnnl::lstm_forward` primitive that processes the entire sequence internally. This eliminates O(T) kernel launches, where T is the sequence length, significantly reducing overhead. The implementation, located in `aten/src/ATen/native/mkldnn/xpu/RNN.cpp`, handles weight layout transformation from PyTorch's `[4*H, I]` format to oneDNN's `ldgoi` order, manages weight reordering, and allocates scratchpad memory. The PR also extends `use_mkldnn()` to return true for XPU in inference mode for float, bfloat16, and float16 data types, and adds packed sequence unwrapping logic for uniform batch sizes (common in batch=1 inference). Correctness was verified against a CPU reference with a maximum difference below 1e-6 across multiple configurations.
Performance measurements on Intel PVC hardware with a bidirectional LSTM (hidden size 256) show dramatic improvements: LSTM inference latency dropped from approximately 34 ms to just 5 ms—a 6.8x reduction. In the end-to-end Kokoro TTS pipeline, this translates to a 40% improvement, from 1.06 seconds to 0.635 seconds. The optimization is specifically for XPU inference; CUDA and CPU paths remain unchanged. A dependency exists on `torch-xpu-ops#3770` for the fused cell fallback path (bias fix). This change enables faster real-time sequence modeling on Intel accelerators, benefiting applications like speech synthesis, natural language processing, and time-series forecasting that rely on LSTM networks.
- Replaces per-timestep fused cell with dnnl::lstm_forward, eliminating O(T) kernel launches for full sequence processing
- LSTM latency drops from ~34ms to ~5ms (6.8x) on Intel PVC; end-to-end Kokoro TTS improves 40% (1.06s→0.635s)
- Correctness verified with max diff < 1e-6 vs CPU reference; only XPU inference affected (float/bf16/fp16)
Why It Matters
PyTorch users on Intel XPU get 6x faster LSTM inference, enabling real-time speech and sequence applications.