An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models

A new study from researchers Hanrui Luo and Shreyank N Gowda reveals a critical flaw in how we test large language models (LLMs) for safety. Published on arXiv, their empirical analysis demonstrates that current jailbreak detection methods—which typically evaluate a single model output—systematically underestimate how vulnerable models like GPT-4 and Claude are to generating harmful content. By testing against the JailbreakBench Behaviors dataset and using multiple sampling techniques, the researchers found that generating and checking 5-10 different responses from a model exposes significantly more harmful behavior than a single check.

The key finding is that moving from single-generation evaluation to moderate multi-sample auditing (5-10 samples) yields the most significant improvement in detection rates, with diminishing returns after that. The study evaluated both lexical TF-IDF detectors and generation-inconsistency detectors, finding that the former often confuses topic-specific cues with actual harmful intent. Furthermore, their cross-generator experiments showed that detection signals can partially generalize across different model families, suggesting a more standardized auditing approach is possible. This research provides a practical blueprint for developers and auditors to more reliably estimate true model vulnerability.