TGO-IV reveals how Transformers reorganize data across layers

A new arXiv paper from Kaustubh Kapil and Kishor P. Upla introduces the Transformer Geometry Observatory TGO-IV, a framework for studying how Transformer representations develop across the forward pass. Rather than inspecting isolated layers, TGO-IV treats each layer's token embeddings as a point cloud and applies persistent homology—a topological data analysis method—to map the birth and death of geometric structures (loops, voids, connected components) across scales. By constructing Vietoris–Rips simplicial complexes, the method captures global shape changes that standard geometric metrics miss. The framework outputs a suite of topological observatories: persistence diagrams, barcode diagrams, Betti curves, persistence landscapes, and bottleneck and Wasserstein distances.

The key contribution is a developmental view: TGO-IV tracks how raw input representations reorganize into task-relevant features layer by layer, potentially isolating critical transition points. This addresses a core interpretability question—'How does a Transformer learn?'—by showing when and where semantic abstraction emerges. For practitioners, this could inform model pruning, layer selection for fine-tuning, or architecture design by identifying which layers do the heaviest representational lifting. The approach is domain-agnostic, applying to both NLP and vision transformers. While the paper is methodological, it opens a promising direction for topological interpretability in deep learning, complementing attention-based and probing-based analyses.