PyTorch adds autograd support to TokenSwitch for efficient MoE training

PyTorch has merged a significant update to its TokenSwitch module, which handles token routing in mixture-of-experts (MoE) models. The commit, authored with assistance from Claude, adds full autograd support to the two core operations—dispatch and combine—that are fundamental to MoE layer execution. These operations are adjoint: the backward pass of dispatch calls combine, and the backward of combine calls dispatch. Previously, users had to implement custom backward functions or rely on non-differentiable routing, limiting the ability to train sparse MoE layers end-to-end with gradient descent.

The new API, designed to mirror conventions like `nn.Module` and `torch.matmul`, provides two usage modes via an optional `out` parameter. When `out=(out_tokens, out_weights, out_idx)` is supplied, the operations write directly to pre-allocated buffers and return them without tracking gradients—ideal for inference or memory-constrained scenarios. When `out` is omitted, internal buffers are allocated and autograd is automatically enabled through wrapper classes `_DispatchAutograd` and `_CombineAutograd`. Developers extending TokenSwitch can implement the raw buffer-writing methods `_dispatch` and `_combine` in subclasses, and inherit both modes from the base class. Notably, `topk_weights` receives no gradient, preserving the routing decision as metadata at a different byte width. This update paves the way for more efficient and flexible training of large-scale sparse models.