Data quality beats model capacity in LoRA closed-book QA study

A new paper from Joan Figuerola Hurtado, posted on arXiv, systematically investigates how to best internalize documents into a language model using LoRA adapters for closed-book question answering. The author fine-tuned a 4-bit Gemma-4-e4b model across roughly 100 training runs, baking in documents ranging from a single file to a 99-document corpus. The core finding: once adapter capacity is sufficient (i.e., LoRA rank grows with corpus size), the single most important lever is training-data quality — far outweighing rank, learning rate, or even alternative architectures. A telling experiment: applying just one data curation pass (shortening gold answers to canonical 1–6 word spans and removing trivial questions) lifted closed-book accuracy from 57.7% to 85.7% on a 15-document set. That improvement dwarfs any architectural change tested.

The paper also reveals practical pitfalls: capacity acts as a hard gate below which no data intervention helps, and there's a subtle coupling between LoRA rank and learning rate that the author initially misdiagnosed. Perhaps most striking, the internalized-LoRA approach beats standard retrieval-augmented generation (RAG) baselines. On the 15-document corpus, the adapter achieved 84.2% recall, outperforming a BM25-RAG pipeline with a base reader (58.9%) and even a 'gold-chunk' oracle that assumes perfect retrieval (65.6%) — all at lower inference latency. The author presents the work as a case study in debugging LLM training empirically, documenting three misdiagnoses along the way. For practitioners, the lesson is clear: when building domain-specific QA systems, spending time on data cleaning and answer formatting may yield far more value than scaling model capacity or hyperparameter tuning.