New study reveals VLMs flip answers under repeated questioning

Researchers from Northeastern University have introduced Just Keep Prompting (JKP), a multi-turn evaluation framework designed to measure the epistemic stability of Vision-Language Models (VLMs) when users repeatedly challenge, question, or contradict their answers. The framework pits models through up to 10 follow-up turns using three adversarial strategies: Adversarial Negation (repeated rejection), Pure Socratic Interrogation (calls to reassess certainty), and Context-Aware Socratic Summarization (reflecting prior rationale before asking for reconsideration).

Evaluating GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B on a subset of the STAR benchmark across 720 multi-turn runs, the study found that while aggregate accuracy changed modestly from Turn 0 to Turn 10, trajectory-level analysis revealed substantial instability: correct answers regressed, wrong answers recovered, and many runs exhibited repeated answer flipping. The effect was strongly model-dependent: Qwen3-VL-30B achieved the highest final accuracy but became confidently wrong under direct contradiction; Gemini 2.5 Pro was comparatively stable but token-expensive; GPT-4o was the most brittle and oscillatory. The authors conclude that multi-turn VLM evaluation captures not just additional reasoning but pressure-response profiles, exposing how models trade off visual grounding, calibration, and conversational compliance under repeated challenge.