Researchers propose CoCo to demystify AI reward models
New CoCo method reveals how 'expert' AI models really make decisions with 40% better clarity.
A new paper proposes CoCo (Contribution Contrast), a method for interpreting Mixture-of-Experts (MoE) reward models at the response level. Instead of relying only on routing weights, CoCo uses chosen-rejected response pairs with the largest contribution contrasts to capture both routing and preference behavior. In automatic and human evaluations, CoCo produced more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives, while maintaining competitive reward modeling accuracy. The authors describe it as the first systematic study of interpretation methods for MoE reward models.
- CoCo is the first systematic method to interpret Mixture-of-Experts reward models by analyzing *response-level* contributions, not just routing weights
- Outperforms prior methods in interpretability by 40% in human evaluations while maintaining competitive accuracy
- Open-sourced by researchers from Saarland University, Max Planck Institute, and Meta
Why It Matters
CoCo could make AI reward models more transparent and trustworthy—critical for deployment in high-stakes applications like healthcare and finance.