Learning from Biased and Costly Data Sources: Minimax-optimal Data Collection under a Budget

A new research paper from Michael Harding, Vikas Singh, and Kirthevasan Kandasamy tackles a fundamental problem in machine learning and statistics: how to optimally collect data from multiple, biased, and costly sources under a fixed budget. The work, titled 'Learning from Biased and Costly Data Sources: Minimax-optimal Data Collection under a Budget,' establishes that common strategies like trying to 'match' a target population distribution are highly suboptimal.

The core technical contribution is a principled sampling plan designed to maximize the effective sample size, defined as the total sample size divided by (D_χ²(q||p̄) + 1), where q is the target distribution, p̄ is the aggregated source distribution, and D_χ² is the χ²-divergence. This plan is paired with a classical post-stratification estimator. The researchers provide an upper bound for the estimator's risk and, crucially, matching lower bounds, proving their approach achieves the budgeted minimax optimal risk. This means it's provably the best possible strategy within the defined theoretical framework.

The implications are significant for real-world data collection in fields like medical studies or political polling, where data comes from heterogeneous sources with different costs and group compositions (e.g., demographics, health markers). The framework also extends to prediction problems for minimizing excess risk. This provides a rigorous, mathematically grounded alternative to ad-hoc data collection methods, promising more efficient use of research budgets and more accurate statistical estimates from imperfect data sources.