Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuning

Reinforcement fine-tuning (RFT) has become a core paradigm for post-training LLMs, but the process remains highly fragile, often requiring expert-driven manual inspection when training goes wrong. Existing efforts address reliability at the system level or tweak algorithms for specific issues, but systematic failure management at the training-process level has been largely unexplored. To fill this gap, researchers led by Lingzhe Zhang created RFT-FaultBench, the first dedicated benchmark for fine-grained RFT failures. The benchmark catalogs 5 fault families (e.g., reward hacking, diverging policies) and 16 distinct fault types across 779 real training runs, logging 22,549 step-level and 1,457,288 trajectory-level records. Their comprehensive empirical study reveals that RFT failures are both observable from training dynamics and distinguishable via empirical fault fingerprints, creating a clear structure for automation.

Building on these insights, the team proposes RFT-FM, an automatic failure management framework that unifies three components in a closed loop: anomaly detection, failure diagnosis, and auto remediation. RFT-FM monitors training metrics in real time, identifies abnormalities using learned patterns from RFT-FaultBench, pinpoints the fault type, and applies targeted corrective actions—all without human intervention. Experimental results show that RFT-FaultBench is neither trivial nor saturated: it exhibits clear anomaly structure while still posing substantial challenges, especially for subtle fault settings. RFT-FM demonstrates strong capability in detecting, diagnosing, and mitigating failures across the benchmark, marking a significant step toward robust, self-healing LLM post-training pipelines.