QuechuaTok reveals tokenizer metrics fail agglutinative languages, PRPE hits 83% accuracy

Tokenization is critical for NLP but standard metrics like fertility rate ignore morphological correctness, especially for agglutinative languages where words are built from many morphemes. The QuechuaTok paper from arXiv (June 2026) by Maria Contreras systematically evaluates four tokenization strategies—BPE, Unigram LM, WordPiece, and a morphology-aware PRPE tokenizer—on Southern Quechua (quz), a low-resource language spoken by 8–10 million people across South America. Using a 200k-sentence corpus and the SQUOIA finite-state morphological analyzer as a silver standard, the study introduces morphological boundary accuracy (MorphAcc) as a necessary metric alongside fertility and OOV rates.

Results are striking: BPE achieves the lowest fertility rate (1.636 at 16k vocabulary) by simply memorizing surface word forms, yet it scores only 6.67% MorphAcc—meaning it almost never splits at correct morpheme boundaries. In contrast, the morphology-aware PRPE tokenizer achieves 83.33% MorphAcc, the highest across all systems, while maintaining reasonable fertility (2.057). This demonstrates that optimizing for fertility rate alone, as many NLP pipelines do, can produce tokenizers that are actually harmful for downstream tasks on agglutinative languages. The authors make all code and models publicly available, urging the community to adopt MorphAcc as a standard evaluation metric for such languages.