New arXiv paper reveals method to detect LLM distillation with near-perfect accuracy

Model distillation—training a smaller model on outputs from a stronger one—has become a common practice to boost performance, but it raises fairness and policy concerns. A new paper from Rajat Rawat and colleagues (arXiv:2607.09692) proposes a method to detect such distillation by comparing a model's outputs to its earlier checkpoint. The technique uses reference-based membership inference, measuring how strongly a student model aligns with outputs from candidate teachers relative to a reference checkpoint from its own lineage. It also infers hidden prompt templates and identifies a distinctive glyph-level signal unique to OpenAI's o1/o3 models. The method is evaluated through a hybrid approach combining controlled distillation experiments with real-world models, addressing the challenge that modern lineages are already heavily entangled.

The method achieves near-perfect accuracy in single-teacher distillation scenarios, even when the underlying pipeline details (such as prompts) are largely unknown. The researchers also introduce statistical tests for both teacher attribution and distillation detection, and extend the framework to open-world settings where no teacher is guaranteed to be among the candidates. Applying the method to contemporary models yields new evidence of potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS. With 27 pages, 21 figures, and 16 tables, this work provides a robust forensic capability for auditing AI model provenance, enabling regulators and developers to detect policy violations and ensure fair competition in the LLM landscape.