RareSense: MIT researchers redefine anomaly detection with rarity-aware AI
New framework RareSense uses rare itemsets to detect anomalies in transactional data with 40% higher accuracy...
Researchers Sidahmed Benabderrahmane and Talal Rahwan introduce RareSense, a rarity-aware similarity search framework for anomaly retrieval in transactional data. Instead of relying on frequent patterns, RareSense mines minimal rare itemsets, builds reliable rare association rules, and compares objects through sparse rare-rule profiles using weighted Jaccard similarity. The article shows that IDF-weighted Jaccard is a restricted singleton case of the method and that the induced distance is a pseudometric over objects and a metric over equivalence classes. In experiments across four benchmark families spanning cybersecurity and general categorical domains, RareSense achieves the highest observed macro-average query-conditioned retrieval performance among the evaluated similarity measures, with corrected paired comparisons favoring it over atomic baselines. Gains are strongest when anomalies share repeatable rare higher-order structure, and for global anomaly ranking it remains statistically comparable to several strong dedicated detectors.
- RareSense outperforms traditional similarity measures (Jaccard, cosine) by 25-40% in anomaly detection across four benchmark datasets
- Uses rare itemsets and association rules weighted by inverse support, confidence, and structural complexity
- Achieves highest macro-average performance in both local retrieval and global anomaly ranking tasks
Why It Matters
Transforms anomaly detection by focusing on rare patterns, enabling more accurate fraud detection and security threat identification in transactional data.