TraceCLIP recovers local semantics from CLIP without training, boosting segmentation by 4.5 points
Unlocks spatially localized semantics from CLIP's global representations without extra training.
TraceCLIP addresses a fundamental limitation of CLIP: while CLIP excels at aligning global image-text representations, it struggles with dense vision-language tasks like object localization and semantic segmentation that require pixel-level understanding. Existing solutions often rely on extra supervision, external models, or fine-tuning, but TraceCLIP is fully training-free. It works by isolating the patch-specific contributions that are written into CLS attention outputs, converting these into a semantic-geodesic topology gate that calibrates final-layer patch affinity for dense feature reconstruction.
The results are striking: on eight zero-shot semantic segmentation benchmarks across multiple backbones and background settings, TraceCLIP improves average mIoU by 1.3 to 4.5 points over the strongest prior training-free methods. This is achieved without any additional training, external vision foundation models, or region-level supervision. The paper suggests that spatially localized semantics remain accessible within CLIP's internally constructed global representations—a finding that could reshape how researchers approach dense vision-language understanding without adding computational overhead.
- TraceCLIP recovers latent patch-level semantics from CLIP's CLS attention output without any training or fine-tuning.
- Achieves 1.3–4.5 point mIoU gains on eight zero-shot semantic segmentation benchmarks over prior training-free methods.
- No external vision models or region-level supervision required, only the internal structure of CLIP's attention.
Why It Matters
Enables high-quality zero-shot segmentation without extra training, reducing compute costs for vision-language grounding applications.