BPE Tokenization Exploit Bypasses LLM Safety with 80% Success

A new paper on arXiv (July 2026) from researchers Li, Liu, and Wu reveals a fundamental safety vulnerability in large language models: the BPE (Byte Pair Encoding) tokenizer. By introducing character-level perturbations that fragment safety-critical words into sub-word tokens, the attack bypasses alignment guardrails while keeping prompts human-readable. The team tested the mechanism end-to-end on five model families (Qwen-3-4B, Qwen-2.5-7B, Gemma-3-4B, Llama-3.1-8B, Mistral-7B) and achieved a 80-100% flip rate on first-token refusal for HarmBench prompts. Critically, 48% of those flips produced genuinely harmful outputs (per-model range 29-65%), with ROC-AUC scores between 0.66-0.98 (pooled 0.84) for distinguishing gap vs. behavior. Activation patching localized the disrupted signal to the last ~30% of layers, and a scan of 30,000 alignment examples found zero fragmented prompts (positive-control recall ≥99%).

On the defense side, the researchers tested 68 DPO configurations across three model families (Qwen-3-4B, Qwen-2.5-7B, Gemma-3-4B) but found no configuration that achieved seed- and pool-stable ASR (Attack Success Rate) closure. SFT (Supervised Fine-Tuning) on fragmented prompts closed ASR on 3/5 families, but only through global collapse — raising refusal rates on benign prompts as well, indicating the missing distribution is necessary but not sufficient under LoRA-16. The paper introduces Conv-Benign, a diagnostic to distinguish selective repair from collapse. All ASR claims are 3-judge-calibrated with stable rankings across judges (±18pp). The findings suggest that current safety training ignores tokenization-level attacks, and a defense requires new data distributions rather than more training.