Cross-Tokenizer LLM Distillation through a Byte-Level Interface

A team of researchers has introduced Byte-Level Distillation (BLD), a novel approach to the challenging problem of cross-tokenizer distillation (CTD). CTD involves transferring knowledge from a large 'teacher' language model to a smaller 'student' model when the two use different tokenizers—the systems that break text into processing units. Existing methods rely on complex heuristics to align mismatched vocabularies. BLD sidesteps this by operating at a more fundamental, shared layer: the byte level. The method converts the teacher's output distribution into byte-level probabilities and attaches a lightweight byte-level decoder to the student model, enabling distillation through this common interface.

Despite its conceptual simplicity, BLD performs competitively with, and in some cases surpasses, significantly more sophisticated CTD techniques. The researchers validated BLD across a range of distillation tasks using models from 1 billion to 8 billion parameters. Their results, detailed in the arXiv paper 'Cross-Tokenizer LLM Distillation through a Byte-Level Interface,' suggest that the byte level is a natural and effective common ground for knowledge transfer. However, the paper also notes that consistent improvements across all tasks remain elusive, underscoring that CTD is still an open research problem. This work provides a strong, simplified baseline that could streamline future efforts in model compression and specialization.