Researchers reveal ImageNet-1k's dirty dozen labels
12% of ImageNet-1k labels are wrong, and it's been this way since 2012
Computer vision researchers from multiple institutions (Illia Volkov, Nikita Kisel, Tetiana Mishkina, Klara Janouskova, Jiri Matas) published a damning analysis of ImageNet-1k, the most cited benchmark in visual recognition since 2012. Their paper 'Doomed to Re-Annotate, Forever: The ImageNet Story' reveals that 12% of original labels are incorrect, 33.3% of images are multilabel (requiring multiple labels), and 3.8% contain no object from any ImageNet class.
The team created ReImageNet, a comprehensively re-annotated version that includes multilabel corrections, object localization, revised class definitions, and semantic attributes (e.g., text-recognition, crowd presence). Using a human-LLM collaboration pipeline with iterative refinement, they achieved substantial accuracy improvements: supervised models gained up to 1.2% top-1 accuracy, while multimodal LLMs (MLLMs) saw 5-6% gains. The structural issues in ImageNet-1k propagate to derivative test sets, proving the problem is systemic rather than isolated.
- ReImageNet reveals 12% of ImageNet-1k's original 2012 labels are incorrect, discovered through human-LLM collaboration
- Revised annotations include multilabel corrections and semantic attributes, boosting supervised models by 1.2% and MLLMs by 5-6% top-1 accuracy
- All 25-page analysis, 16 figures, 8 tables, code, and annotations released publicly on arXiv under cs.CV
Why It Matters
Flawed benchmarks distort AI progress—this cleanup could redefine computer vision evaluation standards and model training practices.