SCALMU fuses physics and deep learning for better hyperspectral imaging
A new unrolled network beats state-of-the-art by embedding learnable matrices into CNMF updates.
SCALMU (Synthetically-trained Coupling of Adaptive Learned Multiplicative Updates) tackles the long-standing trade-off in hyperspectral-multispectral fusion: classical methods like CNMF are physically interpretable but underperform compared to deep learning. The architecture unrolls CNMF's multiplicative updates and replaces fixed steps with learnable matrices, keeping nonnegativity constraints and physical consistency. This hybrid design allows SCALMU to leverage structural priors while benefiting from data-driven optimization.
To overcome the scarcity of real paired hyperspectral-multispectral data, the team generates a synthetic dataset using the dead leaves model—a natural image texture model—and trains SCALMU end-to-end with supervision. On multiple real-world datasets, SCALMU achieves superior fusion quality, both quantitatively and qualitatively, over methods like Deep Unfolding and traditional CNMF. The code is publicly available on GitHub, enabling reproduction and extension.
- Integrates learnable matrices into classical CNMF multiplicative updates, preserving physical interpretability and nonnegativity.
- Trained on synthetic data from the dead leaves model, overcoming real training data scarcity.
- Outperforms state-of-the-art deep learning and classical methods on multiple hyperspectral fusion benchmarks.
Why It Matters
SCALMU offers a principled way to combine physics and learning for high-resolution spectral imaging, enabling practical use in remote sensing and medical imaging.