RAGAL: Fully local RAG assistant runs on a single 8GB laptop
No cloud, no API calls, just a laptop and 25,000 support tickets – RAGAL works.
Dan Musetoiu presents RAGAL, a retrieval-augmented assistant built for the technical support team of AFIR (Romanian Agency for Financing Rural Investments). The system operates under three hard constraints: zero data egress (no external API calls, not even for synthetic data), a read-only mandate (the assistant drafts responses but cannot execute actions), and a single 8 GB consumer laptop as the only development and inference machine. Over a Romanian-language corpus of roughly 25,000 chunks — comprising 15,073 resolved support tickets and internal normative documents — Musetoiu demonstrates that the highest-leverage improvements came from retrieval engineering and retriever fine-tuning rather than a larger generator.
Key technical contributions include hybrid dense-sparse retrieval with intent routing, which raised the internal evaluation score from 62% to 81%. Fine-tuning the bge-m3 embedder on real ticket data improved recall@10 from 0.663 to 0.850 (MRR from 0.489 to 0.684) after just 72 minutes of training. The paper also documents a general pitfall: single-domain fine-tuning silently degraded retrieval performance on untouched document domains, which was detected only through per-domain evaluation and repaired using locally generated queries (GenQ). Two counter-intuitive findings are reported: PII masking improved generation quality, and a structural "anchor distillation" scheme made SQL hallucination impossible by construction. Finally, since zero egress rules out cloud-based evaluation, Musetoiu describes a substitute: a 744B-parameter model run on CPU, too slow for interactive use but affordable as an overnight batch judge. The sanitized pipeline scripts are released for other institutions facing similar data-locality constraints.
- RAGAL runs entirely on a single 8GB consumer laptop with zero external API calls, meeting strict government data residency requirements.
- Hybrid dense-sparse retrieval with intent routing boosted evaluation scores from 62% to 81%; fine-tuning bge-m3 on real ticket data improved recall@10 from 0.663 to 0.850 in 72 minutes.
- Uses a 744B-parameter model on CPU as a local batch judge for evaluation, since cloud judges are prohibited under the zero-egress mandate.
Why It Matters
Demonstrates that enterprise-grade RAG with tight data security is achievable on consumer hardware, opening doors for sensitive government and healthcare deployments.