New EAACD method cuts LLM hallucinations by contrasting expert groups in MoE models
MoE LLMs get a decoding fix that outperforms all baselines on four datasets without retraining.
Mixture-of-Experts (MoE) LLMs are gaining traction for their efficiency, but they still suffer from hallucinations. Existing mitigation methods—prompt engineering and model optimization—either fail to alter internal knowledge or lack cross-domain generalization. Contrastive decoding, which uses layer-wise differences in transformer models (e.g., GPT), has shown promise. However, researchers from multiple institutions found that layer-wise differences do not exist in MoE models with shared experts. Instead, they discovered that higher layers exhibit distinct expert activation patterns between factual and non-factual outputs. This insight led to EAACD (Expert-Aware Adaptive Contrast Decoding), a novel method specifically designed for MoE architectures.
EAACD works by splitting high-layer experts into a higher-reliability group and several lower-reliability groups based on confidence and consistency. It contrasts the higher-reliability group's predictions with each lower-reliability group's predictions to calibrate the model's original outputs. To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking, providing stronger negative references. The method was tested on four QA datasets and outperformed all baselines, including standard contrastive decoding and prompt-based methods. Accepted at ACL2, EAACD offers a lightweight, training-free approach to reducing hallucinations in MoE LLMs, making it immediately applicable to production systems.
- EAACD targets MoE models specifically, filling a gap left by prior contrastive decoding research focused on transformers.
- It leverages distinct expert activation patterns in higher layers to separate factual from non-factual outputs.
- The method splits experts into high/low-reliability groups and uses amplified hallucinations from low-reliability experts as negative references.
- EAACD outperformed all baseline methods across four QA datasets, including standard contrastive decoding and prompt engineering.
Why It Matters
A lightweight decoding method that reduces hallucinations in MoE LLMs without expensive retraining or prompt engineering.