Hugging Face's The Stack v3: 114TB open code dataset for AI training

Hugging Face has unveiled The Stack v3, the largest publicly available code dataset to date, weighing in at a massive 114 terabytes. The dataset is designed to power the next generation of code-generating AI models and is available in two distinct flavors: stack-v3-train and stack-v3-full. The stack-v3-train version is a polished, ready-to-use corpus that has undergone near-deduplication, quality filtering, and PII (personally identifiable information) redaction. The contents are stored inline, so users can simply point the load_dataset function at it and start training without any additional preprocessing. This version is ideal for teams that want to get started quickly with a clean dataset.

The alternative, stack-v3-full, is a raw full dump of the entire 114 TB corpus stored as a Hugging Face Storage Bucket. It retains every duplicate, complete with cluster IDs, and includes stubs for files that were excluded in the filtered version. This gives researchers the freedom to roll their own deduplication, filtering, and mixing strategies, offering maximum flexibility for advanced experimentation. By releasing both versions, Hugging Face caters to a wide range of users: from those who want a turnkey solution for training code models, to those who need full control over data curation. The Stack v3 is expected to drive significant advances in code generation, code understanding, and software engineering automation.