New evaluation method exposes massive precision gap in Earth-observation classifiers

Researchers from the Internal Waves Service (Pinelo et al.) discovered a critical evaluation flaw in operational Earth-observation classifiers. Using Sentinel-1 radar data for detecting internal solitary waves—a rare event occurring in roughly 1 in every 20 scenes—they found that standard balanced-test precision scores wildly overstate real-world performance. A model reporting 0.794 precision on a balanced validation set plummeted to just 0.192 when deployed operationally. This gap, they argue, is an evaluation problem disguised as a training issue: reporting at the wrong class prior creates a systematic artifact invisible to common metrics like F1 or AUC.

The authors propose a prior-matched reporting method based on three numbers: balanced-test precision (for model comparison), operational-prior precision (the rate observed during deployment), and real post-deployment precision (from a sealed, single-read lockbox validation). They then applied a precision-first development cycle, holding recall at a floor of 0.80, and only promoting changes that exceeded a pre-registered margin. After iterating over feature aggregation (the only lever that materially improved performance), the final model achieved 0.927 precision at the operational prior—a 4.8x improvement over naive reporting. The paper highlights that calibration and added model capacity were inert; the honest measure is the contrast between the three numbers. This method generalizes to any operational Earth-observation service bootstrapping a rare-event detector, forcing practitioners to report what the user actually encounters.