PyTorch speeds up CPU quantile/nanquantile up to 8x with partial selection
Full sort replaced by O(n log K) partial selection, 8x faster for median on 10M elements
PyTorch has landed a significant optimization for its quantile and nanquantile operations on CPU, replacing the previous full sort with a partial selection strategy. The change, merged in commit `545b05f` (PR #188394), targets the common case where users request only a handful of quantiles (like median, quartiles, or a few percentiles) from large tensors. Instead of sorting the entire reduced dimension (O(n log n)) and discarding most of the work, the new code uses `std::nth_element` passes over an index permutation to directly pluck the needed order statistics (O(n log K), where K is the number of requested ranks). The NaN handling is preserved by ordering NaN values last, matching the sort semantics.
Benchmarks on an Apple M5 show dramatic gains: for a 10M element tensor requesting a single quantile (median), latency dropped from 1308 ms to 170 ms — a 7.7x speedup. For three quantiles, the speedup is 4.2x, and for ten quantiles, 2.3x. Memory consumption improves accordingly: the 16.7M, three-quantile case uses 147 MB peak RSS instead of 385 MB. The fast path is automatically enabled in CPU eager mode when the number of ranks is ≤100 and `num_ranks^2 ≤ L`; it is gated off for tensor subclasses (compile, export, vmap) and for large rank sets. Output is bit-identical to the sort path, and gradients match for distinct values (tied values remain non-unique, as before). The change also benefits nanquantile and batched inputs, and the gradient is validated via `gradcheck` and `gradgradcheck`.
- Replaces O(n log n) sort with O(n log K) partial selection for CPU quantile/nanquantile
- Achieves 2–8x speedups on Apple M5 (e.g., 10M elements, 1 quantile: 1308 ms → 170 ms)
- Reduces peak memory by ~60% (147 MB vs 385 MB on 16.7M elements, 3 quantiles)
Why It Matters
Data scientists and ML engineers get faster, memory-efficient quantile computations on CPU with zero behavioral changes.