ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling

A research team led by Yongda Yu and 11 other collaborators has introduced ConceptRM, a novel framework designed to combat the pervasive problem of alert fatigue in AI-powered systems. In applications where intelligent agents generate overwhelming volumes of alerts—most of which are false positives—users become desensitized and risk missing critical issues. The traditional approach involves training a reflection model using labeled data from user feedback, but this data is notoriously noisy and expensive to clean manually. ConceptRM addresses this by requiring only a small set of expert annotations as anchors, then creating multiple perturbed datasets with varying noise ratios to train distinct models through co-teaching.

The technical innovation lies in ConceptRM's consensus-based approach: by analyzing where multiple independently-trained models agree, the system can reliably identify negative samples (false alerts) from noisy datasets without extensive manual labeling. Experimental results demonstrate substantial improvements, with ConceptRM outperforming state-of-the-art LLM baselines by up to 53.31% on in-domain datasets and 41.67% on out-of-domain datasets. This represents a breakthrough in cost-effective data cleaning for reflection models, potentially enabling more reliable AI monitoring systems across healthcare, cybersecurity, and industrial applications where false alerts have serious consequences.