MHC Interp #1: Previous-Token Heads Become Attention Sinks Under Manifold-Constrained Hyper-Connections

A new interpretability study on DeepSeek's manifold-constrained hyper-connections (mHC) reveals that the architecture fundamentally shifts how attention heads function. Using 781M-parameter models trained with the mhc-lite repository, the author compared a full mHC model, an mHC-lite variant, and a baseline transformer without mHC. The core finding: previous-token heads—normally detected by diagonal stripe scores—become attention sinks with extremely high kurtosis in mHC models. Standard probing fails to locate them; instead, researchers must use ablation and path patching. These heads effectively turn into 'receiver heads' that absorb information from the residual stream.

Other attention head types also shift positions. Induction heads appear in earlier layers in mHC models compared to the baseline, while duplicate heads appear later. The two variants of mHC—full (using Sinkhorn-Knopp for doubly stochastic weights) and lite (using Birkhoff-von Neumann)—behave differently: mHC-lite's three residual streams each output distinct top-1 tokens, whereas all streams in full mHC converge on the same prediction. The logit lens analysis further confirms that mHC alters the residual stream's role in token prediction. These findings have direct implications for mechanistic interpretability, as existing tools for identifying circuits (e.g., for induction or previous-token heads) must be adapted for mHC-based models like DeepSeek v4.