Tiny SLM hits 96% accuracy on invoice categorization, beating LLMs

A new arXiv paper (2608.18033) from researchers Emma Ceccherini, Daniel Lawson, and Anjulika Salhan investigates how small language models (SLMs) can automate invoice categorization into General Ledger (GL) codes—a task requiring nuanced accounting judgment based on purchasing business, vendor, and invoice text. The team trained a fine-tuned SBERT (sentence-BERT) on a single GPU, achieving 96% accuracy on invoice classification, outperforming both a zero-shot LLM and a vendor-identity baseline. They also demonstrated strong generalization with a 0.9 F1 score using roughly 100 client-specific invoices, making in-house SLMs a viable privacy-preserving, cost-effective alternative to cloud LLMs.

The paper adds a geometric analysis of the pre-trained embedding space: it's globally anisotropic but composed of locally isotropic clusters, and those clusters strongly correlate with vendor identity. This geometry explains why SBERT excels on smaller, challenging categories and new clients. Counterintuitively, structured input that helps human readers didn't improve SLM performance. The findings suggest that in-house SLMs can deliver high accuracy while improving data security and interpretability.