AI research: Final prompt captures only 36% of conversation context

Benjamin Tannenbaum's new paper, 'The Prompt Is Not the Query,' challenges the common practice of treating each user prompt as an isolated query in AI-search evaluation. By analyzing 670 English commercial multi-turn conversations and 7,463 public PRISM conversations, the study introduces 'conversation-conditioned request state' as an observable alternative to latent intent. Results show that the final user prompt contains only a median 35.6% of the session's unique user-side content vocabulary (36.4% in PRISM). In 68.4% of commercial conversations and 74.3% of PRISM conversations, the final prompt contains at most half of that vocabulary.

More importantly, transparent rule-based detection found that at least one request-state dimension present in the conversation history was missing from the final prompt in 50.3% of commercial and 44.8% of PRISM conversations. Among conversations that carried at least one dimension, only 26.1–26.2% reproduced the full set in the final prompt. Meanwhile, 17.9–19.3% added a previously unseen dimension, showing the endpoint is often another state update rather than a summary. These findings strongly support session-level measurement over isolated prompt evaluation for AI search systems, though the study does not estimate causal effects of history on model answers.