New GHK-inspired cache controller closes 72% of fairness gap in LLM serving

Multi-tenant LLM serving puts wildly different request types—short system prompts, long documents, code context, and conversation history—into one shared cache eviction pool. Standard flat-LRU policies handle this poorly: the worst-served class's miss ratio ends up as an uncontrolled fixed point, degrading latency and cost for some tenants. Matt R. Flax's new arXiv paper (2608.14561) attacks this as a multi-class cache fairness problem, introducing a policy class that treats cache promotion like ion flux across a cell membrane.

The controller uses three structural commitments: a single global token-mass imbalance signal, K parallel rectified per-class promotion accumulators, and an age-ordered eviction backstop. Flax instantiates this with a linear V-coupled rectified flux and a Goldman-Hodgkin-Katz (GHK) extension—the same equation family that models ion channels in neuroscience—with the linear form as the GHK limit as voltage approaches zero. On synthetic multi-class workloads across four skew levels, the policy closes 27–72% of the gap between LRU and the theoretical Belady optimum for worst-class miss ratio, with linear and GHK variants statistically interchangeable. The fairness-versus-throughput tradeoff collapses to a single tunable hyperparameter axis.

Flax positions the work as a novel blend of two existing lineages: the LeCaR feedback-controller approach and formal-control-theory cache-decay policies. Unlike those, his controller explicitly targets multi-class fairness rather than aggregate hit rate. For practitioners running inference servers that mix prompts, documents, and chat logs, this offers a practical, tunable knob to prevent tenant starvation without sacrificing overall cache efficiency. Code and reproduction scripts are included in the paper.