New Way to Store Massive AI Data Could Speed Up Your Next Breakthrough
This could make AI training faster and cheaper, benefiting everyone from researchers to consumers.
HPC users often download datasets on HPC systems for analysis and processing. When these datasets are comprised of hundreds of thousands (or more) of small files, accessing them in parallel can put a tremendous burden on the metadata servers of central parallel file systems.
If the dataset is read-only, an effective solution is to compress the dataset into a SquashFS image. There are, however, a number of workflows that can be used to create such a dataset image, and we observed that some were more efficient than others.
To provide more effective guidance for our userbase, we benchmarked eleven different workflows for creating SquashFS dataset image files of datasets with thousands to millions of very small files. We took into account whether the data arrived as tar.gz, zip, or files-and-directories downloads, whether the image creation was on the central file system or local compute node storage, and the number of SquashFS file compression threads used. This paper shares the results of these benchmarks and provides recommendations to HPC system users toward the best workflows to use for SquashFS image creation.
- SquashFS compresses huge AI datasets into one file, making them faster to read and move.
- This speeds up AI training, so new AI features and products can reach you sooner.
- It's mainly for big research computers, but the benefits could eventually reach your devices.
Why It Matters
Faster, cheaper AI training means better AI tools and services for you, sooner and at lower cost.