PyTorch's dynamic test generation: Why CI failures show unfamiliar names

PyTorch's test infrastructure is built for scale, automatically expanding a single test template across multiple devices and dtypes. When a contributor writes a test method with device/dtype parameters, `instantiate_device_type_tests()` generates concrete classes like `TestMatmulCPU` and `TestMatmulCUDA` at import time. The generated method names follow the pattern `<ClassName><DEVICE>.<method>_<device>_<dtype>`, which is why CI failures show names like `test_basic_cuda_float32` instead of the source `test_basic`. This dynamic generation, combined with OpInfos (operator metadata) and CI sharding, allows PyTorch to validate thousands of combinations without manual repetition, but it also causes confusion for new contributors who try to run tests by source name and get “no tests collected.”

To debug effectively, contributors should use `pytest -k` with partial matches of the generated test name (e.g., `-k "test_basic_cuda_float32"`). The `test/run_test.py` script is also optimized for local reproduction of CI failures. Understanding the hierarchy—template classes, device-specific bases, and OpInfo-driven parameterization—is key. For testing their own projects, developers should use public APIs like pytest and `torch.testing.assert_close` instead of PyTorch's internal `torch.testing._internal` helpers. This guide demystifies the naming puzzle and helps contributors focus on fixing bugs rather than navigating test infrastructure.