Foundational Study on Authorship Attribution of Japanese Web Reviews for Actor Analysis
A new study finds simple TF-IDF models can outperform BERT for tracking hundreds of online authors.
A team of researchers from Japan's National Institute of Information and Communications Technology (NICT), including Hiroshi Matsubara and Masaki Hashimoto, has published a foundational study on using AI for authorship attribution, a technique critical for tracking threat actors online. The research, a precursor to analyzing dark web forums, tested four different AI methods on a dataset of Japanese product reviews from Rakuten Ichiba to see which could best identify an author based on their unique writing style.
The study compared a traditional method (TF-IDF with logistic regression) against three modern deep learning approaches: BERT embeddings with logistic regression, fine-tuning the full BERT model, and a metric learning technique. While the fine-tuned BERT model achieved the highest accuracy for smaller groups of authors, its performance became unstable when scaled to several hundred authors. Surprisingly, the simpler, more computationally efficient TF-IDF+LR method proved superior for large-scale analysis, excelling in accuracy, training stability, and cost.
Further analysis revealed key challenges for AI in this field. The primary causes of misclassification were the presence of generic 'boilerplate' text, the model confusing an author's topic focus for their writing style, and reviews that were simply too short to provide a distinctive stylistic fingerprint. The research also demonstrated the practical utility of a 'Top-k' evaluation, where the AI provides a shortlist of most likely authors, which can significantly aid human analysts in screening candidates.
- TF-IDF+LR outperformed BERT for large-scale author sets, offering better accuracy and stability at lower computational cost.
- The study identified boilerplate text, topic dependency, and short text length as the three main causes of AI misclassification.
- This foundational work on clear web Japanese reviews is explicitly aimed at enabling future threat actor analysis on dark web forums.
Why It Matters
This research provides a scalable, cost-effective AI method for cybersecurity professionals to track and identify malicious actors across online platforms.