Gemma-2-2B’s residual stream encodes time in just 31 directions

Researcher Fodenthal presents a preliminary experiment on the residual stream of Gemma-2-2B, revealing a surprisingly structured geometry for tracking information across token positions. By computing the autocorrelation timescale for thousands of directions (random, PCA, and time-lagged probes), they find that the timescale distribution is heavy-tailed: random directions have a median timescale of 1 token, meaning they carry no sequential signal. In contrast, directions discovered by a time-lagged component analysis have a 90th-percentile timescale of 17 tokens. Moreover, only about 31 such directions carry 80% of the total timescale excess, and they are nearly orthogonal to each other, forming a low-dimensional subspace. A crucial control—shuffling token positions within documents—reduces the top-decile timescale from 17 tokens to 1, ruling out unigram frequency or token co-occurrence as explanations; the persistence is truly due to sequential order.

Further analysis shows that projecting out these 31 slow directions from the residual stream almost completely destroys the timescale of held-out probes (Q90 drops from ~19 tokens to effectively 1 token). This is evidence that all long-range contextual information in the residual stream resides in that small subspace—removing it leaves essentially no persistent signal. The finding challenges the intuition that contextual information is diffuse across high-dimensional activation space. Instead, it concentrates in a handful of interpretable directions, potentially enabling targeted interventions to control memory or attention in transformers. Fodenthal notes that this is preliminary work on a single layer and model, and invites replication and debate. The results could have implications for understanding in-context learning, long-context models, and mechanistic interpretability.