Meta's AI Fix Could Make Your Next Phone Cheaper and Faster

Inductor can sometimes create two temporary GPU buffers even when an operator uses the same tensor for both inputs. A new change lets both inputs use one prepared buffer, and the author reports lower peak memory in a workload that uses this pattern. The problem: some operators require their inputs to have the exact strides recorded during tracing, and Inductor prepares each argument position independently, so preparing the same tensor twice can produce two GPU buffers holding the same values with the same strides, both kept live for the operator call.

The fix makes constrain_to_fake_tensors remember inputs it has already prepared while processing one operator's arguments; when it sees the same compiler input again with the same required strides, it returns the prepared result instead of preparing it again. Different IR objects, including distinct views, and different stride requirements still get separate preparation. Reuse works across positional arguments, keyword arguments, and nested containers.

This bookkeeping exists only while compiling one operator call and stores compiler IR references; each subsequent operator gets a fresh cache because the source may have been mutated. The new config.cache_layout_constrained_inputs flag defaults off and can be enabled before compilation with TORCHINDUCTOR_CACHE_LAYOUT_CONSTRAINED_INPUTS=1. Testing: 7 tests passed with GPUs hidden (6 GPU tests skipped as expected), all 13 passed on B200, and earlier B200 checks passed 16 cases for _scaled_mm, its out variant, _scaled_addmm, and its in-place variant. Full CI and end-to-end training remain unverified, and full-model peak memory and throughput gains still require end-to-end measurement.