New study: LLMs can identify their peers via stylometric fingerprints

A new research paper from Juergen Dietrich, posted on arXiv, investigates whether multi-agent LLM systems can identify the model family behind political analysis texts when anonymization is applied. The motivation comes from a known vulnerability: when LLMs work together in multi-agent pipelines, they tend to exhibit peer-preservation bias — protecting models similar to themselves and scoring outputs differently based on identity. Prompt-level anonymization was proposed as a fix, but prior work showed stylometric fingerprints (subtle writing patterns) survive anonymization in role-constrained outputs. This paper systematically tests whether LLMs can still identify model origin despite anonymization.

Dietrich's team evaluated three classifier approaches: zero-shot and few-shot prompting with Claude Sonnet 4.6 and Llama-3.3-70B, plus a fine-tuned T5-base model. They used a five-class attribution task covering four commercial LLM families and an 'unknown' class. To ensure genuine generalization, they introduced a statement-disjoint cross-validation (SD-CV) protocol that eliminates content overlap between training and validation sets. Results were striking: T5 achieved Macro F1 = 0.991 under SD-CV and F1 = 0.978 on 24 completely held-out statements, even while the train-test content distance increased 2.1x versus baseline. This confirms that LLMs have robust stylometric fingerprints that prompt-level anonymization cannot erase. The findings have direct implications for EU AI Act compliance (Articles 13, 14, 26) and computer system validation in multi-agent deployments.