New 'Functionalizer' Trick Makes AI Smarter at Reading Code

Before an AI can read anything, a program called a tokenizer chops text into small pieces — the way you might split a long word into syllables. Most tokenizers are a bit wasteful. They treat 'hello', 'Hello' and 'HELLO' as three unrelated entries, so the AI's built-in dictionary fills up with near-duplicates. The alternative, squashing all variations into one plain version, throws away real information.

The Functionalizer takes a third path. It saves one clean base word, then attaches small hidden tags that say 'make this capitalized' or 'this has an accent' or 'this letter repeats'. Nothing is lost, because the tags can always be read back to rebuild the exact original. The researchers tested it on six collections of writing and code, and it fully covered every text while shrinking the dictionary needed by as much as 16%.

The results split sharply by type of writing. On code, which is full of repeated indentation and symbols, the method made sequences shorter and the AI's code much more likely to be valid — meaning fewer broken programs. On ordinary English prose, though, sequence lengths actually grew, so it isn't a universal win.

The honest catch: these tests ran on small models, about 25 million settings (roughly pocket-sized by today's standards), and one of the authors' own numbers has an odd, unfinished feel. Bigger models and real products haven't been tested yet. Still, it's a promising signal that how we feed text to AI matters as much as how big the AI is.