[D] Lossless tokenizers lose nothing and add nothing — trivial observation or worth formalizing?
Formal proof shows tokenization adds no inherent redundancy, but practical models leak 0.5-2% probability.
A new formal analysis by researcher Douglass Wang tackles a fundamental question in large language model architecture: does the tokenization process inherently limit what models can learn? Using information theory, Wang proves that lossless tokenizers—like the Byte Pair Encoding (BPE) used in GPT-4 and Llama 3—are theoretically sufficient. Through a 'canonical construction,' any target probability distribution over text strings can be exactly represented by a distribution over token sequences without adding extra entropy. This means the tokenization step itself doesn't introduce unavoidable redundancy or restrict expressiveness.
However, the research reveals a crucial practical divergence. While concentrating all probability mass on canonical tokenizations is theoretically optimal, real-world models behave differently. Studies like Chirkova et al. (2023) show models actually leak approximately 0.5% to 2% of their probability onto alternative, non-canonical tokenizations of the same text. Counterintuitively, deliberately introducing this noise through techniques like BPE-Dropout can enhance model generalization. This creates an interesting tension where the mathematically perfect approach isn't always best for building performant AI systems, highlighting how practical engineering considerations sometimes trump pure theoretical optimality in machine learning.
- Formal proof shows lossless tokenization doesn't limit LLM expressiveness or add inherent redundancy
- Real models leak 0.5-2% probability to non-canonical tokenizations despite theoretical optimality
- BPE-Dropout noise improves generalization, creating tension between theory and practice
Why It Matters
Clarifies fundamental limits of tokenization, informing better LLM architecture decisions and training techniques.