Study reveals top code clone detectors rely on shortcuts, not semantics

A team led by Zhiwei Xu from Tsinghua University conducted the first systematic empirical study to evaluate whether state-of-the-art (SOTA) semantic code clone detectors truly understand program semantics or merely exploit dataset-specific patterns. The paper, posted on arXiv, introduces a clone operator framework with eight transformation operators derived from Type-2 and Type-3 clone variations—such as renaming variables, reordering statements, and altering control structures. Using these operators, the researchers generated distribution-shifted yet semantically equivalent Type-4 clone instances from the real-world BigCloneBench dataset. They then evaluated 11 representative detectors spanning token-based (e.g., SourcererCC, Deckard), tree-based (e.g., TBCNN, ASTNN), and graph-based (e.g., CodeBERT, GraphCodeBERT) paradigms.

The results were striking: all tested detectors experienced substantial performance drops when faced with the distribution-shifted clones, despite their impressive benchmark scores. Further analysis revealed that the detectors heavily rely on shortcut learning—they leverage surface-level lexical and structural cues (like identifier names or syntax tree pattern frequency) rather than robust semantic understanding. For example, simple renaming or reordering could fool models that otherwise performed well on standard benchmarks. The study concludes that current SOTA semantic code clone detectors exhibit limited generalizability in real-world scenarios, where unseen variations are common. This highlights critical gaps in evaluation protocols and points to the need for more rigorous testing frameworks that challenge models beyond standard splits.