New study reveals 'Power of Noise' in RAG is fragile under varied settings
Irrelevant documents sometimes help RAG — but new research shows it's easily broken.
A team of researchers (Mazuryk et al.) set out to verify the 'Power of Noise' effect — the surprising finding that adding irrelevant documents to a retrieval-augmented generation (RAG) system can improve question-answering accuracy. Under the original experimental framework (older LLMs, restrictive prompting, constrained decoding), they successfully confirmed the effect. But the real story emerged when they began varying the setup.
By swapping in newer models, adjusting instruction phrasing, or relaxing output length limits, the noise benefit could be made to appear, weaken, or even disappear entirely. Further error analysis showed that much of the original positive result came from truncation and malformed generations. The authors conclude the effect is not a general benefit of noisy retrieval and highlight how fragile inference design choices can be in RAG studies.
- Original 'Power of Noise' effect replicated only under the exact restrictive conditions of the 2024 Cuconasu et al. paper.
- Changing prompt formulation or decoding constraints caused the noise benefit to vanish in many cases.
- Truncation and malformed output accounted for much of the apparent gain, not intrinsic noise usefulness.
Why It Matters
RAG practitioners should not assume adding random documents improves performance — careful tuning of inference is critical.