Your LLM Calls Are Wasting 100 Tokens — Here's How to Get Them Back Without Losing Quality

A new paper from Abhinit Sen, Ajeet Kumar, and Manaranjan Pradhan introduces SPSD (Sentiment Preserving Semantic Distillation), an edge-based prompt compression technique designed to reduce cloud LLM inference costs. The method exploits the 'social-semantic gap' — the observation that many user prompts contain social scaffolding (politeness markers, apologies, repetition) that carries low information value for machine reasoning but consumes input tokens. SPSD runs a 4-bit quantized Small Language Model (SLM) on the user's device (Gemma-2-2B-Instruct) to distill the prompt before transmission to a cloud LLM (Llama-3.1-8B-Instruct).

In evaluation on a 248-prompt corpus, every distilled call yielded positive token savings, with a mean reduction of 99.9 tokens per call. Response quality was assessed using blind LLM-as-judge scoring across 121 pairs on a 15-point rubric. Results showed non-inferiority within a pre-specified 1-point margin: 43% ties, 28% distilled wins, and 29% raw wins. Cosine similarity between distilled and raw responses averaged 0.682 (median 0.712), with 54.1% above the 0.70 threshold. Safety-critical prompts are routed through rule-based gates to bypass distillation. Per-call net energy savings are estimated at 70–270 μWh under stated assumptions, making SPSD a practical approach for reducing cloud LLM costs while preserving response quality.