FindMyText: Open-source tool detects copyrighted text in large web corpora
New fingerprinting method finds near-verbatim copies, outperforming alternatives on 3 datasets.
FindMyText, an open-source Python package from researchers at the Norwegian Computing Center and other institutions, tackles the challenge of detecting copyrighted text in massive web-crawled datasets. The tool builds on traditional document fingerprinting but introduces a novel chain detection mechanism that identifies sequences of matching fingerprints. This allows it to distinguish near-verbatim copies—where text is reproduced with minor alterations—from mere topical similarities. The system uses a distributed, disk-based indexing framework to scale to datasets containing billions of documents, making it practical for real-world applications like auditing large language model training data.
Benchmarked against prior methods on three datasets (ArXiv papers, Wikipedia articles, and generic web content), FindMyText consistently outperformed alternatives in precision and recall. The creators also released a new benchmark for evaluating text containment methods, filling a gap in the field. FindMyText is available on GitHub and PyPI, and the underlying research is detailed in arXiv paper 2607.10020. This tool could become a standard for verifying whether copyrighted material appears in training corpora, helping content creators and regulators enforce IP rights in the age of large-scale AI training.
- FindMyText uses a novel chain-of-fingerprints mechanism to detect near-verbatim copies, not just topical similarity.
- Scalable to web-crawled datasets via distributed disk-based indexing, handling billions of documents.
- Outperforms alternative methods on three benchmarks: ArXiv, Wikipedia, and generic web content.
Why It Matters
Enables robust copyright auditing of AI training data, providing a practical tool for compliance and IP protection.