New Test Checks Whether Your AI Assistant Knows When to Speak Up

AI assistants increasingly promise to remember your conversations — your preferences, your projects, your past decisions. But remembering is only half the job. A new paper proposes TWIST, a standard test for the other half: whether an AI knows when to bring something up, when to quietly fix a draft, and when to stay silent. The test covers four situations, including spotting conflicting information and checking outgoing messages against what you previously said.

To build a fair test, the author had two people independently label 161 examples without seeing the "answer key," then compared notes. They agreed closely — a statistical measure called kappa came out at 0.85, which researchers treat as strong agreement. Crucially, every "catch the mistake" item was paired with a trick example that looks similar but is perfectly fine. That stops an AI from gaming the test by flagging everything as a problem.

The results expose a real trade-off. Systems that simply search past conversations caught 76% to 97% of genuine contradictions — but wrongly flagged 16% to 43% of safe drafts, depending on which underlying model they used. A system built to keep conversations coherent went the opposite way: it almost never cried wolf, but caught only 42% of real contradictions. No tested setup did both well.

Why it matters: an assistant that over-corrects nags you about things you already resolved; one that under-corrects lets you send emails or make plans based on outdated facts. The catch is that TWIST is a proposal, not a standard anyone must follow, and the findings come from a limited set of test setups. Still, it points at a question every memory-equipped AI will face: knowing when to speak up, and when not to.