PixelUp upsampler boosts vision models by +1.2 mIoU without retraining

Vision foundation models (VFMs) like DINOv2 and CLIP provide strong patch-token-level features but often lose fine-grained detail needed for dense prediction tasks. Existing upsamplers fall into two camps: learnable ones that must be retrained for each encoder, and image-guided methods that inject textural artifacts without true semantic understanding. PixelUp solves this by being completely VFM-agnostic—it requires no retraining or fine-tuning when switching between different vision backbones, making it a drop-in upgrade for segmentation, depth estimation, and other pixel-level tasks.

PixelUp achieves this through a coarse-to-fine chain of windowed cross-attention layers, guided by multi-scale semantic features extracted from the VFM itself. This design lets it recover sharp object boundaries while preserving semantic coherence. In experiments across multiple VFMs, PixelUp delivers state-of-the-art results: an average +1.2 mIoU improvement on semantic segmentation, +0.25 δ1 on NYUv2 depth estimation, and notable gains in training-free open-vocabulary (+1.3 mIoU) and unsupervised segmentation (+0.5 mIoU). Because it works with any VFM out of the box, PixelUp could become a standard preprocessing layer for fine-grained vision pipelines, from autonomous driving to medical imaging, without the usual encoder-specific fine-tuning overhead.