IN2P3 releases massive 2024 workload dataset for scheduling research
44 million jobs, 1,000 users, and 105 TB of RAM data now available...
The IN2P3 Computing Center (Villeurbanne, France), a national research unit of CNRS, has released a comprehensive workload dataset capturing every job submitted during 2024. With 44 million jobs from 1,000 users, it significantly expands the availability of real-world scheduling data. Unlike earlier datasets, this one covers an entire year, includes memory usage per job, and is recent, allowing researchers to study seasonal, monthly, and weekly behavioral patterns in high-performance computing clusters.
The dataset was collected from a cluster that scales to 312 machines supporting up to 46,000 concurrent threads and 105 terabytes of RAM. This level of detail lets researchers simulate scheduling algorithms under realistic conditions over long time windows—something smaller or synthetic datasets cannot offer. The combination of extended time interval, memory metrics, and diversity of provenance makes it a valuable resource for distributed systems and parallel computing research.
- Dataset includes 44 million jobs submitted by 1,000 users over the full year of 2024.
- Combines extended time interval, memory usage data, and recent provenance for realistic workload modeling.
- Cluster maxes out at 312 machines, 46k concurrent threads, and 105 TB of RAM.
Why It Matters
Enables researchers to evaluate scheduling algorithms against real, large-scale, seasonal workload patterns.