CORE prompt compression boosts accuracy 30% on edge devices

In agent-driven question answering, RAG often retrieves noisy, redundant context that inflates prompt sizes and slows inference—especially problematic on resource-constrained edge devices. Existing prompt compression methods rely on auxiliary small language models (SLMs) to estimate context importance, but those SLMs add significant memory and compute overhead. Researchers from multiple institutions introduce CORE (Compression via Orthogonal Residual Extraction), a lightweight two-stage approach that operates without any SLM. Stage one constructs an answer set via named entity recognition (NER) and a clue set via semantic matching. Stage two refines the clue set using an orthogonal residual retrieval strategy and filters the answer set with a spatial proximity-based metric. The combined sets form the final compressed context, dramatically reducing token count while preserving key information.

CORE was implemented on an NVIDIA Jetson AGX Orin edge device and a Huawei Nova smartphone. Within a 2000-token budget, it improved accuracy by at least 30.19% compared to baselines like LLMLingua2, reduced memory usage by at least 50.47%, and delivered a 1.94x speedup on the edge device. The energy savings were even more pronounced on the smartphone: a 95.74% reduction over LLMLingua2, making CORE highly practical for mobile deployments. The method’s architecture is simple enough to run efficiently on low-power hardware while outperforming SLM-dependent approaches, opening the door to more capable on-device AI assistants without cloud reliance.