Scalable Pattern Mining Workflow Boosts HPC Log Analysis for Exascale Systems

Modern HPC environments generate massive volumes of log data daily, but their sheer size and heterogeneous nature (especially text logs) make traditional analysis techniques ineffective. Current workflows struggle to extract patterns that indicate system faults or predict failures. To address this, researchers from Argonne National Laboratory and the University of Illinois at Chicago have published a new paper (arXiv:2607.19143) introducing a scalable pattern mining workflow designed specifically for HPC log interpretability.

The proposed workflow combines advanced pattern-matching and mining techniques, including the Aho-Corasick automaton, to systematically identify frequent log patterns and sequences. By storing these patterns in the automaton and incorporating information about system hierarchy and message priority, the workflow enables automated detection of frequent errors and fault events. It then correlates and clusters identified error sequences with job logs, revealing groups of applications sharing similar or dissimilar error signatures. A case study on an exascale-class supercomputer demonstrates the approach's effectiveness, paving the way for real-time monitoring and more resilient, scalable HPC systems through proactive fault prediction.