New study finds 'in-context collapse' in vision-language models, CircA restores accuracy

Mohammad Rostami's new study, posted on arXiv, challenges a core assumption in vision-language model (VLM) prompting: that adding more image–label demonstrations always improves in-context learning. Instead, the paper documents 'in-context collapse'—a sharp, sometimes catastrophic accuracy drop as demonstrations pile up. This occurs across synthetic classification, natural-image classification, and VQA benchmarks, with some models falling below random chance while still producing well-formed outputs. The effect is graded across an open VLM panel from 0.5B to 11B parameters, and also appears in frontier model Claude Sonnet 4.5, making it a systemic issue rather than a small-model artifact.

Using parameter-matched lesion-and-rescue experiments, Rostami causally localizes the collapse to the vision-language integration pathway — specifically the connector and early/mid layers. A lightweight adapter on this pathway restored remap accuracy from 0.39 to 0.91 at 16 shots, while an equal-capacity adapter on the late readout failed. This leads to the proposed solution, CircA: a one-time 'integration vaccine' trained on a single synthetic task that transfers collapse resistance to entirely unseen task families, lifting accuracy from chance to 0.71 on CIFAR and 0.60 on Fashion-MNIST. The paper also finds that the layers best for in-context integration are not the layers best for weight-based consolidation — the late readout achieves higher accuracy and less forgetting with fewer parameters, offering practical guidance for model design.