Developer Tools

trunk/23ef476d0db26c29874503cd4ab54a85fa671b65: revisit guarding in mark dynamic APIs (#176341) (#181469) (#181469)

Dynamo now respects explicit tensor dimension markings with fast C++ guards...

Deep Dive

PyTorch's latest commit (23ef476) introduces a fundamental fix to how Dynamo handles explicit dynamic dimension markings on tensors. The PR, led by laithsakka, addresses a critical gap: marking APIs like mark_static and mark_unbacked previously had no guards for checking the actual indices, while maybe_mark_dynamic completely ignored changes to weak dynamic indices. This meant users' explicit instructions about tensor dimensions could be silently ignored during graph caching.

The new implementation introduces subset semantics for all dimension marking APIs. When you call mark_dynamic(x, [0]), you're telling Dynamo to ensure dimension 0 is dynamic—but saying nothing about other dimensions. The guard logic now ensures that runtime markings must be a subset of compiled markings; if they're not, a recompile triggers. The guard is implemented as a single C++ check (DIMENSION_DYNAMIC_MARKING_GUARD) with two key optimizations: a fast path checks only a single _has_dynamo_dim_marking boolean flag for the common case of unmarked tensors, reducing per-tensor overhead from 4+ PyObject_HasAttr calls to 1. When the compiled tensor has markings but the runtime tensor doesn't, the guard passes immediately since unspecified means "don't care." This gives users precise control over Dynamo's dispatch behavior while maintaining performance.

Key Points
  • Unified C++ guard (DIMENSION_DYNAMIC_MARKING_GUARD) replaces 4+ PyObject_HasAttr calls with a single boolean flag check for unmarked tensors
  • Subset semantics: runtime markings must be a subset of compiled markings; unspecified dimensions mean 'don't care' and reuse cached graphs
  • Per-attribute verbose failure messages for tlparse debugging when guards mismatch

Why It Matters

Gives PyTorch users precise control over tensor dimension dispatch while reducing guard overhead by 75% for unmarked tensors.