Latent Semantic Manifolds in Large Language Models

A new research paper by Mohamed A. Mabrok, 'Latent Semantic Manifolds in Large Language Models,' provides a groundbreaking mathematical framework for understanding the internal geometry of AI models. The work addresses a fundamental mismatch: LLMs perform computations in continuous vector spaces but must output discrete tokens. Mabrok interprets hidden states as points on a 'latent semantic manifold'—a Riemannian submanifold equipped with the Fisher information metric—where tokens correspond to Voronoi regions partitioning this geometric space.

Central to the paper is the definition of the 'expressibility gap,' a geometric measure quantifying the semantic distortion caused by forcing continuous representations into a finite vocabulary. Mabrok proves two key theorems: a rate-distortion lower bound on this distortion for any finite vocabulary, and a linear volume scaling law for the gap derived via the coarea formula. These theorems establish fundamental limits on how well discrete tokens can represent continuous semantic meaning.

The framework was rigorously validated across six transformer architectures ranging from 124 million to 1.5 billion parameters. The analysis confirmed universal geometric patterns, including 'hourglass' intrinsic dimension profiles and smooth curvature structure. Most notably, the expressibility gap scaled linearly with model size, with slopes between 0.87 and 1.12 and extremely high correlation (R² > 0.985). The research also revealed a persistent 'hard core' of boundary-proximal representations that remain invariant to model scale, providing a novel geometric decomposition of model perplexity.

This work has significant implications for multiple areas of AI development. For architecture design, it suggests pathways to reduce semantic distortion. For model compression, it offers geometric principles for more efficient tokenization. For decoding strategies, it provides mathematical grounding for techniques that navigate the latent manifold. Finally, it contributes to scaling law theory by connecting geometric properties to performance metrics, potentially guiding future model development toward more semantically coherent systems.