arXiv's conformal calibration layer fixes multi-modal prediction gaps, restores 19.5% coverage

Multi-modal regression systems—which combine tabular variables, text, images, or other inputs—often produce unreliable prediction intervals when modalities conflict or one is absent at test time. A single global quantile averages across these regimes, missing the specific pattern of missingness. In a new paper published at COPA 2026, researcher Ilia Azizi proposes a modality-aware conformal calibration layer that addresses this head-on. The layer trains or reuses one predictor per modality, computes a disagreement score from their outputs, and plugs it into split conformal calibration under a strict protocol. Two complementary variants are offered: a continuous disagreement-scaled method that reallocates interval width across examples while preserving marginal coverage guarantees, and a Mondrian (stratified) method that calibrates within predefined groups (e.g., by missing modality pattern) to provide group-wise guarantees under joint exchangeability.

Across four multi-modal datasets, the disagreement-scaled layer matched or improved the marginal conformal baseline in 59 of 60 paired runs for interval continuous ranked probability score (CRPS) and in 52 of 60 for interval width—while keeping empirical coverage near the 95% target. In stress tests simulating missing modalities, mask-matched recalibration recovered up to 19.5 percentage points of coverage in the hardest fixed-mask regime. The result is a model-agnostic, lightweight reliability layer that can be dropped into any multi-modal regression system. For practitioners, this means prediction intervals stay honest even when inputs are incomplete—a key concern for real-world deployment where sensor or data failures are common.