ICML 2026 paper proposes joint training to avoid pitfalls of two-stage learning with privileged data

A new paper accepted at ICML 2026 tackles a classic machine learning challenge: how to use extra information available during training (privileged features) that won't be present at deployment. The standard two-stage approach—first train a model on all data, then use its predictions to train a simpler deployment model—can backfire when the privileged information is weak or noisy, causing the deployment model to inherit errors. The authors introduce Coupled Training, which jointly optimizes both models so the deployment model learns to utilize privileged signals only when they actually improve performance.

The method is backed by theoretical guarantees showing when joint training yields higher prediction accuracy, and the team provides a simple alternating training algorithm suitable for large, high-dimensional models. Experiments on synthetic data and real-world prediction tasks demonstrate that Coupled Training robustly outperforms two-stage baselines, avoiding the accuracy degradation caused by misleading privileged features. With 37 pages and 6 figures, the paper offers both rigorous analysis and practical guidance for practitioners dealing with costly or slow-to-collect training-only measurements.