CILN framework creates realistic label noise benchmarks via controlled corruptions

A team of researchers (Shadman Islam, Agustinus Kristiadi, Mostafa Milani) introduced CILN (Controlled Instance-dependent Label Noise), a benchmark generation framework that creates realistic label noise by corrupting input data rather than relying on imperfect human annotators. In CILN, a diverse voter pool labels corrupted instances, making both the source and severity of ambiguity explicit and controllable. The team built 90 benchmark settings across CIFAR-10, MNIST, and Adult datasets, spanning multiple corruption families and severity levels.

Experiments show CILN benchmarks exhibit genuine instance-dependent noise with diverse confusion structures. On CIFAR-10, CILN can produce label distributions closer to human uncertainty than existing synthetic IDN benchmarks. Importantly, CILN exposes failure modes in popular noisy-label learning methods like Co-Teaching and DivideMix that aren't observed under comparable levels of rater-fallibility noise. This suggests noise structure—not just noise rate—plays a critical role in benchmark difficulty and algorithm behavior, providing a complementary framework for studying noisy-label learning under diverse sources of instance difficulty.