Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

A research team from Metropolia University of Applied Sciences has demonstrated that a small, high-quality dataset can effectively adapt a general-purpose large language model for a specialized, low-resource task. By fine-tuning Meta's Llama 3.1-8B model on a validated corpus of simulated clinical conversations in Finnish, they created a model capable of medical transcription. The evaluation, using a rigorous sevenfold cross-validation, yielded a BERTScore F1 of 0.8230, showing strong semantic similarity to reference transcripts, even though traditional n-gram metrics like BLEU (0.1214) were low.

This work directly addresses a critical gap in global healthcare technology: the lack of AI tools for clinical documentation in languages with limited digital resources. The success with Finnish, a morphologically complex language, suggests the approach could be replicated for other underserved languages. The study highlights that model performance hinges more on dataset quality and domain alignment than sheer size, offering a practical blueprint for developing privacy-focused, domain-specific AI without needing massive, often unavailable, training data.

The findings provide a compelling argument for the feasibility of using open-source, locally deployable models like Llama for sensitive healthcare applications. This approach keeps patient data within secure institutional boundaries, a key requirement for medical AI. The research paves the way for similar projects aimed at reducing the administrative burden on clinicians in non-English-speaking regions, ultimately contributing to better patient care and less physician burnout.