Developer Tools

PyTorch adds oneDNN LSTM primitive for XPU, boosting inference 6x

New optimization cuts LSTM latency from 34ms to 5ms on Intel XPU

Deep Dive

This PyTorch pull request (viable/strict/1780997283) adds onednn LSTM primitive support for XPU inference, leveraging Intel's oneAPI Deep Neural Network Library (oneDNN). The core change replaces the existing per-timestep fused cell approach—which sequentially iterates over each time step with individual kernel launches—with a single `dnnl::lstm_forward` primitive that processes the entire sequence internally. This eliminates O(T) kernel launches, where T is the sequence length, significantly reducing overhead. The implementation, located in `aten/src/ATen/native/mkldnn/xpu/RNN.cpp`, handles weight layout transformation from PyTorch's `[4*H, I]` format to oneDNN's `ldgoi` order, manages weight reordering, and allocates scratchpad memory. The PR also extends `use_mkldnn()` to return true for XPU in inference mode for float, bfloat16, and float16 data types, and adds packed sequence unwrapping logic for uniform batch sizes (common in batch=1 inference). Correctness was verified against a CPU reference with a maximum difference below 1e-6 across multiple configurations.

Performance measurements on Intel PVC hardware with a bidirectional LSTM (hidden size 256) show dramatic improvements: LSTM inference latency dropped from approximately 34 ms to just 5 ms—a 6.8x reduction. In the end-to-end Kokoro TTS pipeline, this translates to a 40% improvement, from 1.06 seconds to 0.635 seconds. The optimization is specifically for XPU inference; CUDA and CPU paths remain unchanged. A dependency exists on `torch-xpu-ops#3770` for the fused cell fallback path (bias fix). This change enables faster real-time sequence modeling on Intel accelerators, benefiting applications like speech synthesis, natural language processing, and time-series forecasting that rely on LSTM networks.

Key Points
  • Replaces per-timestep fused cell with dnnl::lstm_forward, eliminating O(T) kernel launches for full sequence processing
  • LSTM latency drops from ~34ms to ~5ms (6.8x) on Intel PVC; end-to-end Kokoro TTS improves 40% (1.06s→0.635s)
  • Correctness verified with max diff < 1e-6 vs CPU reference; only XPU inference affected (float/bf16/fp16)

Why It Matters

PyTorch users on Intel XPU get 6x faster LSTM inference, enabling real-time speech and sequence applications.

📬 Get the top 10 AI stories daily