FID fails Earth observation: study shows metrics misalign with humans and downstream tasks
FID scores swing wildly on rotated images, but humans see no difference.
A new study from Umit Mert Çağlar and Alptekin Temizel systematically evaluates how well popular data-quality metrics align with human judgment and downstream task performance in Earth observation. The researchers compared real satellite imagery with synthetic counterparts generated by deep generative models, using metrics like FID, KID, IS, LPIPS and SSIM. Their core finding: these metrics, especially FID, are poor proxies for both human perception and real-world utility when applied to geospatial data.
Specifically, the team showed that semantics-preserving augmentations—such as rotating an image—caused FID scores to change dramatically, while human subjects consistently recognized the same land cover. In another test, synthetic samples that scored poorly on automatic metrics were rated as equally or more realistic than real images by human evaluators. When these low-scoring synthetic images were mixed with real data to train semantic segmentation models, downstream performance actually improved. The authors conclude that automatic quality evaluation for synthetic geospatial data should be grounded in task performance and human evaluation, not in metrics derived from ImageNet-pretrained feature spaces.
- FID, KID, LPIPS, and SSIM all misalign with human perception when applied to Earth observation images—rotation changes metric scores but not human judgment.
- Synthetic images with poor FID scores were rated as equally or more realistic than real data by human evaluators, and improved segmentation when combined with real images.
- Study recommends grounding synthetic data quality evaluation in downstream task performance and human assessment, not ImageNet-derived metrics.
Why It Matters
For AI teams generating synthetic remote-sensing data, relying on FID alone can reject useful training samples, wasting data and compute.