PARED: New inverse RL method aligns AI by imitating human demonstrations
No preference annotations needed – just demonstrations for AI alignment.
Aligning large language models typically relies on supervised fine-tuning from demonstrations or reinforcement learning with human feedback. A new paper from researchers at Carnegie Mellon and Apple introduces PARED (Projected Alignment Reward Estimated from Demonstrations), which tackles alignment using inverse reinforcement learning. PARED recovers an implicit reward function from expert demonstrations by training a lightweight discriminator that separates demonstration samples from the policy's own samples in a feature space. This explicit reward can be inspected, reused, and optimized on-policy without requiring expensive preference annotations.
Experiments show that PARED improves a base policy both at inference time (via reranking) and through on-policy RL after supervised fine-tuning. The method also supports contextual alignment, allowing a single policy to adapt to different audience preferences by adjusting the reward function. By leveraging demonstrations alone, PARED reduces the cost of alignment and opens the door to more flexible, reusable reward signals. The paper demonstrates that inverse RL can serve as a practical alternative to traditional reward modeling for AI safety and instruction following.
- PARED uses inverse RL to extract an explicit reward function from expert demonstrations without task-specific preference annotations.
- The method enables both inference-time reranking and on-policy RL optimization, improving base policies without supervised loss.
- Supports contextual alignment, letting a single policy be tailored to different audience preferences by reweighting reward features.
Why It Matters
Simplifies AI alignment by leveraging demonstrations alone, reducing reliance on expensive human feedback for safer, more adaptable models.