New NLP backlog tool detects security-relevant items at F2=0.774

Engineering teams in regulated domains struggle to spot security requirements hidden in backlog items, which are often written as brief, free-form descriptions. A new paper, accepted at ASE 2026 Industry Showcase, tackles this with an automated enrichment system. Developed by Ignacio García Núñez, Florian Angermeir, and Fabiola Moyón Constante, the approach uses a security-relevance classifier to identify risky backlog items, then a retrieval-augmented generation (RAG) pipeline pulls relevant clauses from security requirements documents. The system is validated with a dataset of 288 backlog items labeled by nine security practitioners, showing substantial agreement (Fleiss' κ=0.787).

On the technical side, the classifier achieves an F2 score of 0.774 in-distribution and mean zero-shot G-measure of ~0.65 across five benchmarks, matching or beating classical ML and open-source GPT baselines. A preliminary evaluation with two practitioners on industrial backlogs used company-internal policies and CIS Benchmarks, retrieving 24 clauses — 12 rated at least 4/5 for relevance. This suggests that NLP-driven backlog enrichment can help engineers surface security requirements early, moving organizations closer to continuous security compliance rather than bolt-on audits.