New ViT prompt tuning method yields gains on 34 datasets

Visual prompt tuning is a parameter-efficient way to adapt large Vision Transformers (ViTs) to downstream tasks by inserting learnable tokens. Traditionally, these prompts are fused with image tokens via concatenation or addition. In a new paper accepted at ECCV 2026, Xiao et al. ask whether a single fusion scheme is optimal. They formulate the problem as a bi-level optimization and leverage differentiable architecture search (NAS) to jointly learn both the prompt content and the fusion operation per transformer layer. To enrich the search space, they propose two additional fusion schemes: affine transformation (a learned linear mapping) and cross-attention (querying prompts from image tokens). The method, called Layer-Specific Prompt Fusion Discovery, automatically selects the best fusion for each layer.

Extensive experiments on 34 datasets spanning VTAB-1k, FGVC, and HTA show consistent gains over VPT-Deep and recent baselines. The hybrid fusion approach achieves favorable accuracy–latency–parameter trade-offs, often improving top-1 accuracy by 1-3% with similar or lower compute. Their analysis reveals that different layers prefer different fusion strategies: early layers benefit from additive fusion, while deeper layers leverage cross-attention. This provides a novel perspective on how prompts interact with image tokens, and the method is generalizable to other transformer-based vision architectures. The authors release code to facilitate further research.