XAI Visualizations Can Cause Over-Trust in Discriminatory Models
Accurate but irrelevant data in XAI can make users trust biased models.
Deep Dive
A new study shows that providing accurate but superfluous data in a model explanation can lead users to unjustifiably trust even highly discriminatory predictive models. In a crowdsourced experiment, participants rated a biased model more favorably when given irrelevant yet accurate data, highlighting a 'trust junk' effect. The paper warns XAI designers to consider the rhetorical power of their work.
Key Points
- Crowdsourced study showed that accurate but irrelevant visualizations increase trust in discriminatory models by up to 30%.
- The 'trust junk' effect occurs when users conflate quality of explanation with model fairness.
- Authors urge XAI designers to test for unintended rhetorical effects of visualizations.
Why It Matters
Visual explanations can inadvertently legitimize biased AI, making critical evaluation of XAI design essential.