PyTorch migrates NVGEMM backend to official cutlass.operators API
New API brings native FP4 support and fixes kernel selection bug
In a significant infrastructure update for PyTorch's GPU kernel generation, the NVGEMM (NVIDIA GEMM) backend has been migrated from the internal alpha package cutlass_api to the official cutlass.operators API. This migration, part of PR #189772, is largely mechanical but includes important renames: get_kernels becomes get_operators, Kernel and KernelMetadata become Operator and OperatorMetadata, and various attribute names are standardized (e.g., kernel_name → operator_name, min_cc → designed_for_min_cc). The new API natively maps torch.float4_e2m1fn_x2, making the previous _ensure_fp4_dtype_registered function effectively a no-op safety net.
The migration also fixed a critical bug in the vendored _supports wrapper for dense_blockscaled_gemm_kernel. The old _infer_scale_swizzle_impl check incorrectly rejected valid arguments on the transposed B operand, leaving zero eligible kernels. Replacing it with the upstream ScaledOperand.numel_scale element-count check now surfaces 60 eligible kernels. This fix had to be bundled because the old check used cutlass_api APIs that were removed in the migration. The PR was authored with an AI assistant (Claude) and approved by PyTorch core reviewers drisspg and crcrpar.
- Mechanical migration from cutlass_api to cutlass.operators: all kernel/operator classes and metadata renamed for API consistency
- Native mapping of torch.float4_e2m1fn_x2 eliminates need for manual FP4 dtype registration
- Critical _supports wrapper fix: _infer_scale_swizzle_impl replaced with ScaledOperand.numel_scale check, boosting eligible kernels from 0 to 60
Why It Matters
Simplifies GPU kernel maintenance and fixes a bug that broke kernel selection for scaled matrix multiplications.