Byte-level LLMs get smarter with EntropyMoE routing

A team of researchers from four institutions—led by Bo Liu, Muxuan Yu, Yu Zhang, Pengfei Gao, and Yongping Zhang—has developed EntropyMoE, a breakthrough Mixture-of-Experts (MoE) architecture for tokenizer-free large language models (LLMs). Unlike traditional MoE systems that rely on fixed token representations, EntropyMoE groups raw bytes into dynamically sized patches and uses patch entropy—a measure of information density—to route computation efficiently. By replacing dense feed-forward layers with Top-K expert layers, the system adapts its computational capacity to the semantic variability of each patch, reducing wasteful processing of low-information segments.

In experiments, EntropyMoE achieved the lowest held-out bits-per-byte (a metric for compression efficiency) compared to matched dense and sparse baselines, while maintaining comparable downstream accuracy. This demonstrates that patch entropy is an effective signal for routing sparse conditional computation, extending MoE modeling beyond traditional tokenizer-based representations. The work, published on arXiv as cs.AI:2608.06398, represents a step toward more adaptive and efficient LLM architectures that operate directly on raw byte sequences.