New research tightens sample bounds for entropy estimation
MIT researchers prove min-entropy needs Θ(k log k) samples, not k/log k
Researchers Arman Adibi and Piotr Krysta from MIT have published a groundbreaking paper that tightens the sample complexity bounds for estimating Renyi and min-entropy, two fundamental metrics in information theory. Their work addresses a long-standing gap in understanding how many samples are needed to accurately estimate these entropies.
The team demonstrates that min-entropy estimation to constant additive accuracy requires Θ(k log k) samples, significantly revising prior assumptions that suggested Θ(k/log k) samples were sufficient. For Renyi entropy of integer order α between 2 and a logarithmic factor of k, they prove matching upper and lower bounds of Θ(α k^(1-1/α)). Their upper bound leverages an unbiased falling-factorial estimator based on α-way collisions, while the lower bound uses a construction where a heavy symbol is hidden in random locations. The results also show that min-entropy can approximate high-order Renyi entropy when α is a sufficiently large multiple of log k, leading to a sample complexity of Θ_ε(k log k) in such regimes.
- Min-entropy estimation to constant accuracy requires Θ(k log k) samples, correcting prior Θ(k/log k) claims.
- For integer orders 2 ≤ α ≤ c₀ log k, Renyi entropy estimation matches Θ(α k^(1-1/α)) bounds.
- High-order Renyi entropy (large α) can be uniformly approximated by min-entropy, reducing sample complexity.
Why It Matters
This research fundamentally reshapes how we estimate entropy in large-alphabet systems, impacting machine learning, cryptography, and data compression.