New test-driven method exposes 2.56x more privacy leaks in LLM code generation

A new study from researchers at multiple universities (including Peking University and Nanyang Technological University) introduces a pipeline to systematically probe privacy leaks in LLM-based code generation. The problem: large language models trained on massive code datasets can memorize and reproduce sensitive personally identifiable information (PII) hidden in the training data. Existing detection methods rely on ad-hoc prompt construction (manually or automatically designed prompts) that fail to mimic how PII actually appears in real-world code contexts, leading to underreported leakage.

The proposed solution uses a test-driven strategy: it generates realistic privacy-related code scenarios and forces the LLM to produce test cases that expose memorized PII. A key innovation is an automatically constructed privacy feature library that provides realistic templates and examples, replacing manual prompt engineering. Large-scale experiments on 5 widely used LLMs show the pipeline achieves a 2.56x increase in detected privacy leakage compared to existing baselines. This work provides a more practical and scalable method for auditing privacy risks in code-generating AI, with implications for developers, enterprise users, and regulators concerned about confidential data exposure from AI assistants.