Latent Reasoning Sprint #4: PCA Analysis on CoDI
Interpretability sprint shows CoDI model forgets its own reasoning steps, potentially limiting scalability.
In a detailed technical sprint posted to LessWrong, independent researcher Realmbird conducted a PCA (Principal Component Analysis) on the CoDI (Chain-of-Dot-Interpretation) Llama 3.2 1B model to probe its latent reasoning mechanisms. The analysis, building on previous work with activation steering, focused on comparing hidden state activations against the model's KV (Key-Value) cache. A key finding was that the first principal component (PC1) of hidden state activations strongly correlates with the model's <|eocot|> (end of chain-of-thought) token across all latent positions in the GSM8K math dataset. This suggests the hidden state encodes a clear signal related to concluding a reasoning step.
However, the analysis revealed a significant architectural critique of CoDI. The model operates by running multiple latent reasoning passes but only retains the final KV cache to generate an answer, discarding the intermediate hidden states. Realmbird describes this as giving CoDI a 'goldfish memory,' where it forgets the detailed reasoning process that led to its conclusion. This design means traditional hidden state steering requires an additional forward pass, while KV cache steering can be applied directly but showed less interpretable structure in the PCA. The researcher speculates that this memory limitation could impede the model's ability to scale for more complex, multi-step reasoning tasks, as it cannot reflect on or build upon its own prior latent computations.
- PCA on hidden states reveals a strong correlation with the <|eocot|> token, providing a mechanistic clue to CoDI's reasoning process.
- The CoDI architecture discards hidden states after latent reasoning, retaining only the KV cache for final answer generation.
- This 'goldfish memory' design may limit scalability, as the model cannot access its full reasoning history for future steps.
Why It Matters
Reveals fundamental trade-offs in AI reasoning architectures, showing how design choices for efficiency can create interpretability and scalability bottlenecks.