Know2Guess Benchmark Exposes When LLMs Should Guess vs Abstain

A team of researchers from multiple institutions has released Know2Guess, a novel benchmark designed to rigorously evaluate whether large language models can reliably answer questions they should know and abstain from those they shouldn't—without conflating performance with data contamination or prompt idiosyncrasy. The benchmark contains 1,200 items across five domains, each with explicit abstention expectations and contamination-risk metadata. It uses dual parsing (strict and normalized) and locked answer-or-abstain prompts to isolate genuine knowledge boundaries from generic refusal behavior.

Testing FLAN-T5, Qwen2.5-Instruct, and Llama-3-Instruct models revealed that no model fully masters the answer-or-abstain task. Qwen2.5-3B-Instruct achieved the best overall reliability, but calibration remains poor and benign-item refusal persists. FLAN-T5 baselines showed weak productive abstention. The benchmark exposes a selective, incomplete transition from answering to abstaining even in strong instruction-tuned models. Prompt and parser robustness analyses preserved rankings, confirming Know2Guess is a usable, reproducible protocol for auditing how LLMs handle knowledge boundaries, with the dataset publicly available.