PRISM framework boosts multimodal AI rubric accuracy by 3x
New PRISM framework trains multimodal models to follow complex, prioritized rules 3x better with just 10K samples
PRISM (Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis), developed by researchers from institutions including Tsinghua University and ByteDance, introduces a four-stage framework to train multimodal models on complex, prioritized instructions. Unlike traditional approaches that treat instruction-following as single-question answering, PRISM casts models as executors that must verify each rule in a rubric before producing judgments.
The framework generates persona-task pairs, prefix-guided rule sets, quality-filtered rubrics, and structured verification traces. PRISM-Eval's deterministic metrics (Loose and Strict) eliminate the need for judge models during evaluation. With just 10K synthesized samples, PRISM lifts Qwen3-VL-4B's strict accuracy from 9.5% to 30.1% on PRISM-Eval while preserving general benchmark performance. The gains transfer to four additional open-source models across dense and MoE architectures.
- PRISM trains models as rule executors rather than single-question answerers
- Qwen3-VL-4B's strict accuracy improved 3.17x (9.5% → 30.1%) using only 10K training samples
- Gains transfer to 4+ other open-source multimodal LLMs with no architecture restrictions
Why It Matters
Enables AI systems to reliably follow complex, prioritized rules in real-world multimodal scenarios like medical imaging or legal document analysis