Beyond Single Reports: Evaluating Automated ATT&CK Technique Extraction in Multi-Report Campaign Settings

A research team from North Carolina State University has published a comprehensive evaluation of automated methods for extracting MITRE ATT&CK techniques from cybersecurity threat intelligence (CTI) reports. The study, led by Md Nazmul Haque and colleagues, tested 29 different AI approaches—spanning named entity recognition (NER), encoder-based classification, and decoder-based LLM methods—across 90 reports from three major attack campaigns: SolarWinds, XZ Utils, and Log4j. Their key finding reveals that analyzing multiple reports together, rather than individually, boosts detection performance significantly, with F1 scores improving by about 26% on average. This aggregation helps security teams build a more complete picture of campaign behavior, as different reports often contain complementary technical details and high-level overviews.

Despite this improvement, the research exposes substantial limitations in current AI capabilities. Performance saturates after analyzing just 5-15 reports, suggesting diminishing returns for additional intelligence. More critically, even the best methods achieved maximum F1 scores of only 78.6% for SolarWinds and 54.9% for XZ Utils, leaving many attack techniques undetected. The study found that up to 33.3% of AI misclassifications involve semantically similar techniques that share tactics, and these errors have a disproportionate negative impact on identifying the security controls needed for defense. Interestingly, longer, more technical reports—despite having lower readability scores—consistently yielded better extraction results, highlighting the importance of detailed source material for AI analysis.