HybridCodeAuthorship benchmark: AI code detection fails on real mixed-code repos
Top detector achieves just 56% F1 score on identifying AI-written lines in real-world Python code.
Rapid adoption of AI code assistants (LLMs like GitHub Copilot, Codex) has made industry codebases a mix of human- and AI-authored code. For risk management and productivity analysis, fine-grained location detection of AI-generated code is critical. However, existing benchmarks use academic, LeetCode-style problems and assume a snippet is entirely human or entirely AI—a false dichotomy in practice. To address this, Patterson, Wang, and Faulkner introduce HybridCodeAuthorship, a novel benchmark dataset of Python files with interleaved human- and AI-authored lines, designed to simulate authentic AI assistant usage.
The dataset was built by leveraging CodeSearchNet, a large collection of open-source GitHub repos. The authors then benchmarked two state-of-the-art AI-generated code detection algorithms at both line and chunk levels. Results: the best algorithm, AIGCode Detector, achieved an F1 score of only 0.48 at chunk-level and 0.56 at line-level—far from perfect. This highlights that even modern detectors struggle with realistic, mixed-code scenarios. Accepted to LREC 2026, HybridCodeAuthorship sets a new, more challenging standard for the emerging field of code authorship attribution.
- HybridCodeAuthorship is built from real GitHub code (CodeSearchNet) with mixed human/AI lines, not simple LeetCode-style snippets.
- Top-performing detector (AIGCode Detector) achieves only 0.56 F1 at line-level and 0.48 at chunk-level.
- Existing benchmarks are overly simplistic (binary human/AI), failing to reflect real-world hybrid codebases.
Why It Matters
As AI assistants write more production code, accurate line-level detection is essential for risk, compliance, and productivity measurement.