Llama, Mistral, Qwen tested: dedicated MT still beats local LLMs

A new paper from Mihael Arcan, "Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation," systematically tests how well local large language models perform compared to dedicated machine translation systems. The study evaluates three instruction-tuned LLMs—llama3.2:3b, mistral:latest, and qwen2.5:14b—against OPUS-MT and NLLB-200 baselines using the FLORES devtest split for nine official European Union languages (English-to-Romance and English-to-Germanic families). The experiments vary prompt scope (single target language vs. JSON-formatted family-scope prompts requesting all languages in a family at once) and demonstration selection (zero-shot, and few-shot with k=5 examples chosen randomly, by lexical similarity, or by embedding similarity).

Results show that dedicated MT systems remain the strongest overall, particularly for Germanic languages. Few-shot prompting improves performance for mistral:latest and qwen2.5:14b but actually harms llama3.2:3b. Embedding-similarity retrieval yields the best average scores for the stronger LLMs, but its advantage over random or lexical examples is modest. Family-scope prompting works well only for the larger, more capable models (qwen2.5:14b and mistral:latest), while smaller models like llama3.2:3b frequently produce structured-output failures when asked to generate multiple translations in a single JSON response. These findings highlight the need to evaluate LLM translation not just by language pair and metric, but also by prompt design, retrieval strategy, and multi-target compliance—critical considerations as developers increasingly use local LLMs as cost-effective alternatives to cloud-based MT services.