Game-theoretic fine-tuning finds optimal KL regularization for LLMs

A new paper from researchers including Michael I. Jordan introduces a game-theoretic approach to post-training LLMs that eliminates the need for heuristic hyperparameter search of the KL regularization coefficient. The framework models fine-tuning as a sequential game: an agent chooses a policy to maximize cumulative reward, while a monitor observes the policy's outputs over time and tests for deviations from a reference policy. The resulting equilibrium policy, the authors show, is equivalent to the solution of a KL-regularized RL problem—but with an optimal regularization coefficient that maximizes reward per unit of statistical distinguishability. This coefficient is learned via concave-convex fractional programming, allowing seamless integration into existing fine-tuning pipelines.

In experiments with Qwen3-8B and Llama-3.2-1B, the method achieved competitive reward-retention trade-offs in a continual learning setting, outperforming heuristic coefficient choices. Notably, the framework also provides a principled way to audit API providers: by monitoring the distinguishability of model outputs, clients can detect whether a hosted model has been fine-tuned beyond agreed-upon bounds. The work thus offers both a practical improvement to fine-tuning efficiency and a new tool for AI governance.