New IGSD method reveals hidden content channels in GPT-2 and Qwen transformers
Standard importance scores miss early-layer content transport—IGSD finds it.
Yifeng Guo, Jin-Hong Du, and Xiang Chen propose Interchange-Group Sobol Decomposition (IGSD) to address a flaw in mechanistic interpretability: traditional importance scores conflate whether a component matters because it transports relevant content or because its removal degrades the computation. IGSD uses matched activation replacement versus zero ablation on the same component, estimating two Sobol-style variance indices and their signed difference. A symmetric off-manifold diagnostic (ST>1) ensures intervention validity.
In factual recall experiments on GPT-2 small (124M parameters) and Qwen2.5-1.5B (1.5B parameters), IGSD reveals early-layer content channels that standard methods miss. A controlled donor design shows early layers transport relation-frame content while late attention heads transport subject-retrieval content, refining to known head Attn_{L9H8}. Late-layer clamping confirms the early signal is expressed through downstream transformations, not residual pass-through. The work demonstrates that replacement and deletion are not interchangeable controls, providing a practical diagnostic for content transport.
- IGSD distinguishes content transport from computation degradation using paired interventions and Sobol indices.
- Applied to GPT-2 small and Qwen2.5-1.5B, IGSD finds early-layer relation-frame channels missed by standard importance.
- The method refines granularity to specific attention heads (Attn_{L9H8}) and reveals downstream expression of early signals.
Why It Matters
Better interpretability of transformer internals leads to safer, more controllable AI systems.