Marivate's annotation paradox: low-resource NLP scaling outpaces human evaluation

Vukosi Marivate's new arXiv paper, "The Annotation Scarcity Paradox in Low-Resource NLP Evaluation," offers a critical narrative survey of the field from 2014 to the present. It identifies a structural friction: while cross-lingual transfer, massively multilingual models, and proliferating benchmarks have driven explosive growth, the sociolinguistic expertise needed to evaluate these generative systems is severely strained, inequitably distributed, and structurally marginalized. The paper conceptualizes this as the Annotation Scarcity Paradox, arguing that the technical capacity to scale models vastly outpaces the sovereign human infrastructure required for authentic evaluation. Marivate traces three phases: early heuristic optimism (2014–2018), the illusions of top-down benchmark scaling (2018–2023), and the current era of generative bottlenecks (2023–present).

Key concerns include extractive data pipelines that plunder under-resourced languages, undercompensated "ghost work" by local language experts, and "language data flaring"—short-lived hype cycles that extract data without building lasting capacity. These practices threaten the epistemic validity of reported progress. The paper surveys emerging responses such as data augmentation, model-based evaluation, participatory curation, and annotation-efficient approaches using item response theory and active learning. It assesses their equity and validity trade-offs, warning that purely technical fixes risk perpetuating the same power imbalances.

Marivate closes with a practitioner call to action, urging a paradigm shift from transactional data extraction to relational, community-embedded evaluation rooted in epistemic governance, data sovereignty, and shared ownership. For AI practitioners and policymakers, this is a timely reminder that scaling multilingual models without investing in local expertise undermines both accuracy and ethics.