TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation

A research team from Yale and collaborating institutions has introduced TAB-PO, a novel method for aligning large language models that specifically targets a major weakness in current techniques. Standard Direct Preference Optimization (DPO) works well for general instruction following but fails in 'token-critical structured prediction' settings, such as extracting hierarchical labels and evidence spans from medical messages. In these tasks, the 'chosen' and 'rejected' AI completions provided for training can be nearly identical, differing by only 1-3 tokens (low-separation pairs), while the meaning hinges entirely on those sparse, high-importance tokens. DPO's sequence-level approach suffers from 'margin collapse,' where it can't properly separate these near-identical outputs, and 'gradient dilution,' where the learning signal gets lost in common structural text like JSON formatting.

The TAB-PO algorithm solves this by augmenting DPO with two key innovations: token-weighted advantages that focus the model's learning on the rare, semantically critical tokens (like a medical code), and a conditional token-level barrier that acts as a regularizer to prevent overconfidence on uncertain predictions. This ensures the model balances staying close to its original supervised fine-tuning (SFT) knowledge while effectively learning from subtle human preferences. Evaluated on the complex task of annotating patient-provider communications, TAB-PO delivered a consistent ~4% relative improvement in micro-F1 score over the SFT baseline and outperformed other recent preference optimization methods. This breakthrough paves the way for more reliable AI in high-stakes domains where precision on specific terms is non-negotiable.