New TACO method fixes RL credit assignment for flawed LLM reasoning

A team of seven researchers from multiple institutions has published a paper on arXiv (2607.07976) identifying a fundamental flaw in how reinforcement learning (RL) is applied to large language models (LLMs). They call the problem 'Positive-Credit Contamination': when using critic-free RL methods like GRPO, uniform credit assignment gives the same positive advantage to all tokens in a trajectory—including low-probability tail tokens that are contextually erroneous. This causes the model to reinforce flawed reasoning behaviors indiscriminately, undermining training quality.

To address this, the authors introduce TACO (Tail-Aware Credit Calibration). TACO first computes a tail-risk score for each token by analyzing local generation context, distinguishing between unexpected rarity (which should not be reinforced) and uncertainty-driven exploration (which may be useful). It then uses this score to calibrate positive credit: suppressing gradients for incidental noise while allowing recurring useful rare patterns to accumulate reinforcement. Experiments across three different LLMs and eight benchmarks show TACO consistently outperforms GRPO-style baselines. Notably, it also improves training stability, enabling sustained performance gains in long-horizon RL tasks. The code is publicly available.