CAGE: Game theory aligns LLMs with multiple objectives at inference

Aligning large language models with human preferences is inherently multi-objective—different users and benchmarks impose heterogeneous, often conflicting requirements. Now, researchers from Baiting Chen, Tong Zhu, Rui Yu, and Xiaowu Dai propose CAGE (Common-Agency Games for Alignment), a game-theoretic framework that handles this challenge at test time without any retraining.

CAGE models each alignment objective (e.g., helpfulness, harmlessness, factual accuracy) as a strategic principal that allocates token-level incentives to a shared LLM agent. The interaction reaches an equilibrium policy—a Nash equilibrium of a common-agency game—that captures the joint effect of competing objectives. The team develops an efficient algorithm based on equilibrium problems with equilibrium constraints (EPEC) to compute this equilibrium, and provides theoretical guarantees including existence, uniqueness, convergence, and no-regret learning dynamics.

Empirically, CAGE enables flexible, fine-grained trade-offs across objectives at inference time, consistently outperforming existing test-time alignment methods like beam search or best-of-N sampling. Notably, it supports weak-to-strong generalization, meaning a small, weak LLM can use the CAGE policy to match the performance of a much larger model—making multi-objective alignment practical in resource-constrained settings.

This work bridges game theory and LLM alignment, offering a principled way to balance multiple preferences without expensive retraining. As models become increasingly deployed in diverse real-world applications, CAGE provides a scalable, training-free solution for aligning outputs with complex, multi-stakeholder values.