New telecom AI study: Best LoRA model had worst validation loss

A new arXiv paper by Tamic, Jaffeux-Cheniout, and Marjou examines parameter-efficient fine-tuning (PEFT) for small language models (SLMs) in telecommunications customer support. The team applied Low-Rank Adaptation (LoRA) to Qwen2.5-3B, a 3-billion parameter model, and generated 30,000 synthetic training examples using a combinatorial method based on 52 telecom-specific terms. The pipeline used Gemini 2.0 Flash to produce 1,560 distinct problem scenarios. They evaluated 16 LoRA configurations by varying hyperparameters and target modules, going beyond standard metrics by incorporating energy consumption analysis and qualitative assessment via an LLM-as-a-judge framework using GPT-5.2 and Claude 4.5 Sonnet.

Results reveal a stark divergence between quantitative and qualitative performance: the configuration with the best validation loss (0.5024) ranked only 6th-7th in human-aligned evaluation, while the worst loss (0.6807) took first place according to both judge models. This demonstrates that validation loss alone is insufficient for selecting fine-tuning configurations in conversational AI. The paper also provides insights into target module selection for LoRA injection and an energy-performance trade-off analysis, contributing to sustainable LLM deployment. For telecoms handling sensitive customer data, this study underscores the need for human-aligned evaluation metrics over traditional loss-based optimization.