LLM context compaction drops 83% of user constraints; plug-in restores them

When context windows run out of room, LLM systems compact earlier conversation history to continue tasks. But a new paper from researchers including Zhiqi Wang and Dongwon Lee reveals a silent failure: instructions meant to govern the entire session—called Session Constraints (SCs), like "do not delete any emails until I confirm"—get dropped during compaction. To quantify this, they built COMPINT, an evaluation suite testing compactors across three long-context scenarios: multi-turn chat, agentic trajectory, and long-horizon research.

Results are stark: current compactors retain only 17% of injected SCs on average, and most perform worse than skipping compaction entirely. Retention varies sharply with the compactor model, prompt wording, context length, SC phrasing, and injection location, showing the loss is systematic. To fix it, the team proposes an SC-aware extractor that runs alongside the compactor as a plug-and-play module, achieving over 90% retention across all three scenarios without modifying the compactor or the underlying LLM. The COMPINT suite and code are open-sourced, giving developers a benchmark to test and improve constraint adherence in long-session AI systems.