Bi-Predictability: A Real-Time Signal for Monitoring LLM Interaction Integrity

Researchers Wael Hafez and Amir Nazeri have introduced a novel framework called 'Bi-Predictability' (P) for monitoring the structural integrity of multi-turn conversations with large language models (LLMs). Published in a March 2026 arXiv paper, their work addresses a critical gap in AI reliability: current methods like perplexity or semantic entropy either work post-hoc, measure only output confidence, or require heavy computational sampling. These fail to detect when an ongoing interaction between a user and an LLM—such as in an AI agent or customer service chatbot—gradually degrades while still producing seemingly coherent individual responses.

The core innovation is the 'Information Digital Twin' (IDT), a lightweight architecture that estimates Bi-Predictability by analyzing raw token frequency statistics across the conversational loop of context, response, and next prompt. It requires no secondary model inferences or embedding calculations, making it highly efficient. In experiments involving 4,500 conversational turns between a student model and three frontier teacher models, the IDT demonstrated 100% sensitivity in detecting artificially injected disruptions. Crucially, the research revealed that structural coupling (the logical flow of conversation) and semantic quality (the perceived goodness of an individual response) are often separate. The IDT's P metric aligned with structural consistency 85% of the time, but with human semantic judge scores only 44% of the time, exposing a regime of 'silent uncoupling' where conversations break down undetected.

This decoupling of structural monitoring from semantic evaluation provides a scalable, real-time signal for AI assurance. For developers deploying LLMs in autonomous workflows, customer support, or tutoring systems, the IDT offers a practical tool for closed-loop regulation. It can trigger interventions—like resetting a conversation or flagging it for human review—when structural integrity drops, preventing gradual failure modes that current evaluation methods miss. This moves beyond judging single outputs to ensuring the health of the entire interactive process.