How Much Data is Enough? The Zeta Law of Discoverability in Biomedical Data, featuring the enigmatic Riemann zeta function

A new theoretical paper by researcher Paul M. Thompson introduces the 'Zeta Law of Discoverability,' a framework for predicting when adding more biomedical data will cease to significantly improve AI model performance. The work addresses a critical bottleneck in fields like medical imaging and genomics, where datasets can scale to millions of samples. Instead of relying solely on empirical trial-and-error, the framework provides a mathematical model based on the spectral structure of data covariance operators and task-aligned signal projections.

The key innovation is that many standard performance metrics, such as Area Under the Curve (AUC), can be expressed as a cumulative accumulation of signal-to-noise energy across identifiable spectral modes. Under common assumptions, this accumulation follows a scaling law governed by power-law decay, which naturally leads to the appearance of the famous Riemann zeta function from number theory. The theory explains how representation learning techniques—like sparse models or contrastive learning—improve efficiency by concentrating useful signal into fewer, more stable modes.

Practically, the Zeta Law predicts 'cross-over' regimes where simpler models outperform complex ones at small sample sizes, but are overtaken once sufficient data stabilizes the additional parameters of a high-capacity or multimodal encoder. This gives researchers and developers a principled method to allocate resources, deciding whether to invest in more data collection, better model architectures, or integrating new data modalities like combining MRI scans with genetic data to accelerate discovery in applications from disease classification to topological data analysis.