LLMs expose synthetic tables: LLaMA misses 100% while Gemini catches them all

Privacy and data sharing are often at odds, especially with tabular data where synthetic tables are used to reduce risk. A new preprint from Manel Slokom, Malek Slokom, and Thierno Kante proposes using large language models (LLMs) as discriminators to classify whether a table sample is real or synthetic. They test two settings: C1 with only the table data, and C2 with added distributional metadata. Using LLaMA as an open-source model and Gemini as a reference, they evaluated three synthesis models (CTGAN, TVAE, Gaussian Copula) on the UCI Adult and ACS Census datasets across 451 trials.

The results reveal dramatic gaps. On Adult, LLaMA achieved a detection rate (DRS) of 0% for reported cells, meaning it failed to identify any synthetic samples. In contrast, Gemini scored 100% DRS for CTGAN and TVAE. On Census, LLaMA predicted 'SYNTHETIC' for most samples, while Gemini remained highly accurate in C1 but dropped performance for CTGAN and TVAE when given metadata in C2. The authors also compared against a classifier two-sample test (C2ST) and record linkage as distributional baselines, and ran a human pilot with 2 annotators and 240 trials to ground truth.

The findings highlight that LLM-based discrimination is a viable privacy audit signal, but model choice, provider reporting, and data encoding must be handled carefully. The paper underscores how synthetic data can still 'look real' to some LLMs, while others are extremely sensitive. For organizations relying on synthetic tabular data, this offers a new way to gauge potential privacy leaks before sharing data externally. Code and experiment scripts are available on GitHub for reproducibility.