Post-Training with Policy Gradients: Optimality and the Base Model Barrier

A new theoretical paper by researchers Alireza Mousavi-Hosseini and Murat A. Erdogdu provides a rigorous mathematical framework for understanding the limits of fine-tuning large language models (LLMs) like GPT-4 or Llama 3. The study focuses on post-training using policy gradient (PG) methods with outcome rewards—the common practice of using human or AI feedback to align a model. The authors prove that while PG is efficient at refining a model's performance on tasks where its base (pre-trained) model already has a non-trivial likelihood (α), it hits a hard wall. This 'Base Model Barrier' means the model's expected error after fine-tuning is fundamentally limited by a property of the original model called its Likelihood Quantile (LQ).

Crucially, the research shows that to push a model's performance beyond the support of its pre-trained knowledge—to correct fundamental gaps or hallucinations—PG with outcome rewards may require an exponential number of reward queries relative to sequence length (N), making it practically impossible. However, the paper also identifies a potential path forward. By shifting from outcome rewards (judging the final answer) to process rewards (judging each step or token), PG variants can depend on a token-level LQ, potentially avoiding this 'curse of dimensionality.' This finding could guide more efficient training strategies for next-generation AI agents that need to reason step-by-step.