LLMs fail at Braille translation: small T5 model outperforms giants
State-of-the-art LLMs produce unstable, poor Korean-Braille translations despite multilingual training.
A new paper from researcher Abdullah Abdullah, accepted at the LTEDI Workshop at ACL 2026, systematically evaluates state-of-the-art LLMs on bidirectional Korean-Braille translation using a human-annotated dataset. The results are stark: even multilingual, instruction-tuned models produce consistently poor, unstable outputs that substantially disagree with human judgments. The authors attribute these failures to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns—a critical gap for accessibility-dependent users.
In contrast, supervised fine-tuning of a small T5-small model on the same data yields large and stable gains over zero-shot and prompted LLM baselines across all standard metrics (SacreBLEU, ChrF++, CER, BLEU, ROUGE-L, METEOR, CIDEr). This demonstrates that modest task-specific supervision can dramatically outperform general-purpose LLMs on low-resource accessibility tasks. The finding challenges the assumption that scaling and instruction-tuning alone can generalize to all modalities and underscores the need for specialized, Braille-aware approaches in assistive AI systems.
- SOTA LLMs fail at bidirectional Korean-Braille translation, producing unstable outputs with high human disagreement.
- A tiny T5-small model fine-tuned on the same data beats all zero-shot/prompted LLM baselines across 7 metrics.
- Failures linked to missing Braille-aware tokenization and weak alignment between Korean and Braille patterns.
Why It Matters
Crucial for accessibility tech: relying on large LLMs for Braille could fail—small specialized models are safer.