Qwen vs Gemma tokenization: 2.6x fewer tokens for code

A Reddit post by user WhoRoger has sparked discussion about why two popular open-weight models behave so differently. Testing identical 330-line HTML/JS code, Qwen 35B A3B tokenized the input into 1,609 tokens, while Gemma 26B A4B produced 4,258 tokens—a 2.6x disparity. The finding suggests Qwen's tokenizer recognizes code structurally, capturing syntax as specialized input/output patterns, whereas Gemma breaks code into word-like pieces similar to natural language. That alone helps explain why Qwen is widely considered superior at coding tasks, while Gemma excels in language-related benchmarks.

Interestingly, the gap nearly vanished on a 55-line instruction document: Qwen used 1,025 tokens and Gemma 1,039, indicating both models tokenize regular text similarly. WhoRoger also noted that Qwen shifts into a distinct reasoning mode when handling code, while Gemma treats it as ordinary language—likely increasing compute overhead and reducing context efficiency. The post references a LiquidAI project exploring retraining existing models with more efficient tokenizers, raising the question: could a better tokenizer help Gemma close the coding gap? For developers, the takeaway is clear—tokenizer design directly impacts model behavior, cost, and task suitability, and efficiency differences like this deserve more attention in model evaluation.