Uncertainty sampling survives label noise, but not on all datasets, arXiv study shows

Active learning promises cheaper labeling by focusing on informative examples, but it assumes labels are reliable. In a rigorous study posted to arXiv, John Myron Uy investigates a core vulnerability: whether margin-based uncertainty sampling degrades under label noise because it acquires more corrupted labels or because errors cluster in hard-to-label regions. Using logistic regression with per-budget cross-validation, the author runs 100 paired seeds across three binary tabular datasets—Breast Cancer Wisconsin, Banknote Authentication, and MAGIC Gamma Telescope—testing noise rates from 0 to 0.30 and annotation budgets from 20 to 120, plus a clean-label extension to budget 400.

The headline result: with clean labels, uncertainty sampling consistently improves normalized balanced-accuracy area under the learning curve by 1.09 to 1.77 percentage points over random sampling on all datasets. Under noisy conditions, however, the picture gets murkier. Difficulty-dependent noise erodes uncertainty sampling's advantage more severely than random classification noise on Breast Cancer Wisconsin at six of eight tested rates—but shows no such penalty on the other two datasets. An exposure-matched control that aligns mean final acquired corruption finds no universal additional penalty from structured error location. Even on clean MAGIC data, uncertainty sampling boosts balanced accuracy while reducing average precision and true-positive rate at fixed false-positive rates. The takeaway: uncertainty sampling is broadly label-efficient, but its robustness is highly sensitive to dataset, budget, noise structure, and the evaluation metric chosen.