Bayesian calibration catches hallucinated packages missed by AI code detectors

A team of researchers from NewCo Partners and Sorbonne Université has introduced a Bayesian calibration layer for slopsquat detectors—tools that flag hallucinated package imports in code produced by large language models. Traditional pipelines output binary decisions (flag or not), but this new layer emits a Beta-posterior probability per detection. The probability is derived from a 3-category epistemic taxonomy that classifies each prior as empirically calibrated, constructively argued, or engineering-judgement-traced. Beyond checking the primary 200/404 registry channel, the calibrated layer exploits PyPI metadata signals—package age, release count, author descriptor, summary—to surface registered-but-suspicious packages that a binary registry detector would miss, which the authors argue is the realistic post-LLM-emission attacker regime.

The calibration is evaluated on a merged corpus of 1,734 Python snippets: a stratified 189-prompt BigCodeBench slice plus a 100-prompt niche-library stress-test set. These prompts were run across six models: Claude-Sonnet-4.6, Mistral-Large, DeepSeek-v4-pro, DeepSeek-R1, Mistral Codestral, and Meta CodeLlama. Against a binary baseline that shares the registry oracle with ground truth (a degenerate upper bound), the calibrated layer reproduces strict-registry detections and adds well-calibrated flags via the metadata channel. Detector asymmetry is assessed with a McNemar paired test, calibration with a flagged-subset Expected Calibration Error and a full-corpus Brier score. The result is a risk-aware primitive consumable by downstream CI gates, supporting principled threshold decisions across detection rules.