References Improve LLM Alignment in Non-Verifiable Domains
New method achieves 73.1% on AlpacaEval with Llama-3-8B, matching performance of specialized reward models.
Deep Dive
Researchers from Yale and Salesforce introduced a method using reference outputs from frontier models to improve LLM-based evaluators for alignment in non-verifiable domains. Their reference-guided approach enhanced less-capable LLM-judges and enabled effective self-improvement. On Llama-3-8B-Instruct, it scored 73.1% on AlpacaEval and 58.7% on Arena-Hard, showing average gains of +20.2 points over standard fine-tuning and +5.3 over reference-free methods.
Why It Matters
Enables cheaper, more effective AI alignment without needing expensive human feedback or specialized reward models.