M2Note lets VLMs learn from mistakes without retraining or weight updates

Vision Language Models (VLMs) still suffer from recurring failures like skipping visual checks or hallucinating concepts. Existing solutions rely on expensive fine-tuning or reinforcement learning that breaks under distribution shift. Enter M2Note (Multimodal Mistake Notebook Learning), a training-free framework that externalizes learning into an editable memory. M2Note transforms failed trajectories into compact subject-guidance notes: the subject captures the domain and concept, while the guidance provides actionable verification steps reusable in future inference. At test time, M2Note retrieves relevant notes via multimodal RAG and appends them to the model context, steering reasoning away from previously observed pitfalls. It also uses batch-level post-verification with rollback to ensure stability, committing edits only if they improve batch performance.

Experiments on six multimodal reasoning benchmarks show consistent improvements across domains and backbones, achieving strong cost and sample efficiency. M2Note supports both self-evolving (same VLM as solver and supervisor) and cross-model evolution (stronger supervisor guides a weaker solver), enabling capability transfer without weight updates. The framework complements Chain-of-Thought prompting and delivers gains without the brittleness of retraining. This approach offers a practical path for continual improvement of deployed VLMs, especially where retraining is impractical or data is scarce.