Poole & Lee's FMR cuts AI misalignment by 98% in imitation learning

AI alignment research often relies on multi-stage pipelines that combine human demonstrations and feedback, typically optimized for contextual bandit settings like language generation. However, extending these methods to fully sequential decision-making—where agents act over time—remains underexplored. Benjamin Poole and Minwoo Lee identify this gap, noting that existing approaches treat demonstrations and evaluative feedback as separate inputs rather than as an interconnected signal. Their work introduces Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that leverages feedback to correct imitation learning policies during offline training. By adapting Safety Gymnasium environments as a testbed for alignment evaluation, the authors demonstrate that FMR reduces misalignment by up to 98% across a range of imitation learning algorithms. The method also excels in limited data regimes, where it learns effectively from scarce aligned demonstrations and even from uninformative noisy data.

FMR's key innovation lies in its regularization approach: instead of fine-tuning with reinforcement learning after imitation, it directly integrates corrective signals from human feedback into the batch training process. This single-stage design avoids the complexity and data inefficiency of multi-stage pipelines. In practice, FMR works with any imitation learning algorithm, making it a drop-in improvement for existing systems. The researchers validated their approach on Safety Gymnasium, a suite of continuous control tasks that test safety constraints such as avoiding obstacles or respecting boundaries. With a 98% reduction in misaligned behaviors, FMR sets a new benchmark for offline agent alignment. For practitioners deploying AI in high-stakes environments—like robotics, autonomous driving, or healthcare—this offers a principled way to align agents with human values using minimal and imperfect feedback, all without requiring online interaction or complex reward engineering.