Scientists Find a Hidden Map Inside AI That Shows How It Thinks
This discovery could open the AI 'black box' and make AI safer to use.
New research finds that transformer models have a "privileged geometry" in their residual streams: the directions nearest a prediction shape the immediate answer, while the next directions decide where the output goes about five tokens later. The work reveals a tightly stratified structure, including temporal stratification of output behavior, and links this to earlier findings that model geometry is affected by the Adam optimizer, which creates a privileged basis. The author argues the evidence supports calling this structure privileged and hopes the findings, code, and preprint on arXiv will start a community discussion.
- AI models have a hidden 'compass' inside them that affects every answer they give.
- The structure was found by looking at directions in the AI's internal memory—closest directions decide the answer type, further ones decide where it goes.
- This could help engineers create AI that is easier to test, control, and trust.
Why It Matters
Understanding how AI thinks is key to making it safer—this research brings us one step closer.