LessWrong research: Approximate natural latents have exact error prices
Exact tradeoffs between redundancy and mediation errors for natural latents discovered.
The post proves that for jointly Gaussian views with correlation less than 1, any latent variable that is exactly recoverable from each view separately (i.e., with zero redundancy error) must be independent of the entire system—a degenerate case. This means non-degenerate exact natural latents are impossible. The key contribution is an exact sum rule derived from the chain rule of mutual information: the mediation error (agreement not explained by the latent) plus the redundancy error (information about the latent inaccessible from one view) equals a fixed quantity determined by the joint distribution. This holds distribution-free.
In the Gaussian case, the optimal latent and its errors can be expressed in closed form via canonical correlation analysis. The result shows that the best approximate natural latent is unique and its approximation errors satisfy a Pythagorean-like identity. This gives AI researchers a precise mathematical handle on the tradeoff between how natural (redundant and mediating) a latent is and its informativeness. The work bridges information theory and natural abstraction, offering exact quantitative guidance for building world models that compress multiple views into a shared representation.
- Exact natural latents exist only as degenerate constants for noisy correlated views.
- A distribution-free sum rule links mediation error and redundancy error: they sum to a constant.
- Jointly Gaussian case yields closed-form optimal latents via canonical correlation analysis.
Why It Matters
Provides exact mathematical foundation for approximate natural abstraction, enabling principled design of shared representations from multiple data sources.