Llama-Server’s KV Cache Bug Wasted 2.5GB Restores – Simple 117-line Sidecar Fix

A detailed investigation by the llama-server team uncovered a critical bug in the slot save/restore feature. Designed to allow long-context sessions to be parked on disk and resumed later without paying the prefill tax, the feature appeared to work flawlessly during a process lifecycle. Restoring 2.49 GB of KV state took just 1.23 seconds. However, the very next completion request would force a full re-prefill (e.g., 17.5 minutes at 100K context), rendering the entire save/restore pipeline useless. The root cause was a single piece of metadata — the checkpoint list (slot.prompt.checkpoints) — that lived only in process memory and was never serialized.

The fix required 117 lines of C++ in tools/server/server-context.cpp. At save time, the checkpoint list is persisted to a versioned sidecar file (<state>.ckpt); at restore, it is reloaded. A previous attempt to synthesize a checkpoint at the tip failed because the necessary rollback targets a position before the tip. With the sidecar approach, rollback lands on a real persisted checkpoint, reprocessing only a few tail tokens. Fallback to tip synthesis handles states saved by older builds. Key metrics: baseline full prefill at 100K took 722.4 seconds; save took 1,777 ms for 2.56 GB.