Multiple Independent Semantic Axes in Gemma 3 270M

New interpretability research on Google's Gemma 3 270M model reveals how small language models internally organize semantic information through multiple independent axes, with abstract/concrete and social/nonsocial emerging as privileged organizational structures. The study, conducted by researcher CharlesL and published on LessWrong, provides unprecedented insight into how 270-million parameter models process and categorize different types of information at the neural feature level.

Background/Context: This research builds on previous work analyzing GPT-2's residual stream, where initial signs of an abstract-social vs concrete-physical axis were detected. However, those earlier findings showed superposed representations that made it difficult to understand what specific features the model was actually tracking. The move to Gemma 3 270M with Sparse Autoencoders (SAEs) represents a significant methodological advancement in AI interpretability, allowing researchers to move beyond simply detecting that prompts are different to understanding the specific feature composition driving those differences.

Technical Details: The study employed Gemma Scope 2 16k SAEs for analysis rather than raw activations, enabling researchers to see activations in terms of interpretable features. The abstract/concrete axis was found to be defined by feature clusters rather than single features, with abstract prompts activating reasoning operation features (like f116 for qualification and f200 for problems) while concrete prompts triggered physical domain ontology features (like f230 for composition and f437 for geology). The separation emerges gradually through processing layers, starting with nearly 50% feature overlap at layer 5, becoming mostly separated by layer 9, and continuing refinement through layers 12 and 15.

The research tested five potential semantic axes: abstract/concrete, social/nonsocial, formal/informal, positive/negative, and animate/inanimate. Abstract/concrete and social/nonsocial showed the strongest separation with Jaccard similarities of 0.102 and 0.083 respectively, significantly outperforming positive/negative and animate/inanimate axes which showed similarities in the 0.28–0.29 range. Surprisingly, these axes use separate representational features with minimal overlap, suggesting independent dimensions rather than overlapping representations of the same underlying concepts.

Impact Analysis: This research has significant implications for AI safety and interpretability. Understanding how models internally organize semantic information could lead to better alignment techniques and more predictable model behavior. For developers working with small language models like Gemma 3 270M, these findings suggest that certain semantic distinctions (abstract/concrete, social/nonsocial) are fundamentally built into the model's architecture, which could inform prompt engineering strategies and fine-tuning approaches. The methodology using SAEs rather than raw activations represents a practical advancement for researchers seeking to understand model internals.

Future Implications: As AI models continue to scale, understanding their internal representations becomes increasingly critical for safety and alignment. This research demonstrates that even relatively small models (270M parameters) develop sophisticated internal organizations for semantic information. Future work could explore whether these findings scale to larger models, how training data influences axis formation, and whether these organizational structures can be deliberately shaped during training. The discovery of independent semantic axes suggests potential pathways for creating more interpretable and controllable AI systems by understanding and potentially influencing these fundamental organizational structures.