Long-context AI beats RAG by 12% but costs 26x more tokens
A new study quantifies the 'token tax' of feeding entire docs into LLMs.
A new paper from researchers at Miami University systematically compares two architectures for document-grounded generative AI: retrieval-augmented generation (RAG) and long-context prompting. RAG retrieves only a few relevant passages before generation, while long-context loads the entire document collection into the model’s context window. The study, submitted to arXiv on June 18, 2026, evaluates 972 answers across three machines, two small language models, and three retrieval/prompting approaches using an expert-validated manufacturing safety benchmark.
The results reveal a clear accuracy–cost trade-off. Long-context prompting achieved the highest correctness (73.1% vs. 65.4% for semantic RAG) but came at 26 times the per-query token cost — a gap the authors call the 'token tax' of broader evidentiary access. The paper emphasizes that for knowledge-intensive, high-stakes work, the choice between RAG and long-context depends on organizational resources and accuracy requirements. Resource-constrained teams may accept lower accuracy from RAG to avoid massive token bills.
- Long-context prompting achieved 73.1% accuracy vs. 65.4% for semantic RAG.
- Per-query token cost was 26x higher for long-context, dubbed the 'token tax'.
- Study used 972 answers across 3 machines and 2 small LLMs on a manufacturing safety benchmark.
Why It Matters
RAG vs. long-context isn't free — accuracy costs 26x more tokens in this benchmark.