HERO benchmark library standardizes federated continual learning evaluation

Federated continual learning (FCL) trains distributed models on non-stationary data without forgetting, but existing evaluations are incomparable due to entangled choices in datasets, task splits, client data splits, and backbones. The HERO library, introduced by Thinh T. H. Nguyen and six co-authors, addresses this by providing a decomposition framework. HERO-Core, the main comparable benchmark, uses α (0 to 1) to control client data skew and ρ (0 to 1) to control task-order mismatch across clients. This allows researchers to isolate specific heterogeneity factors and compare methods fairly. The library includes implementations of representative FCL methods and supports both image benchmarks (CIFAR-100, TinyImageNet) and graph-based domain incremental learning (Domain-IL) on OGB-MolPCBA.

Early results using HERO reveal critical insights: method behavior shifts dramatically between easy and heterogeneous settings; average accuracy alone masks poor performance on the bottom 10% of clients; task-order mismatch rewards different strategies than synchronized evaluation; and the same HERO interface can expose domain-shift difficulty beyond image-based FCL. For example, on OGB-MolPCBA, scaffold-domain granularity changes input distribution while the prediction task stays fixed, a harder scenario than typical image shifts. HERO ships 30 pages of documentation, 10 figures, and full scripts, enabling reproducible and setting-aware evaluations. This library is a significant step toward trustworthy comparisons in the rapidly growing field of federated continual learning.