Meta's OpenZL uses graph model for faster, smaller compression
New compression framework beats general-purpose algorithms while cutting dev time from months to days.
Meta researchers led by Yann Collet (creator of Zstd) have unveiled OpenZL, a new compression framework that fundamentally rethinks how compression algorithms are designed and deployed. OpenZL introduces a "graph model" that represents the compression pipeline as a directed acyclic graph (DAG) of modular codecs, each handling a specific transformation. This architecture allows engineers to compose application-specific compressors by wiring together pre-built components, then compresses data into a self-describing wire format that any universal decoder can decompress. The result is a system that matches or beats the compression ratios of specialized general-purpose compressors while also being faster on a variety of real-world datasets from Meta's production environments.
Compared to advanced deep-learning-based compressors that achieve high compression ratios at enormous computational cost, OpenZL is competitive in ratio but runs orders of magnitude faster. Internal deployments across Meta showed consistent size and speed improvements, and the time to develop a new application-specific compressor dropped from months to just days. This breakthrough addresses a long-standing industry problem: application-specific compressors outperform generic ones but are notoriously difficult to build, maintain, and scale. By modularizing the process and providing a universal decoder, OpenZL makes practical, high-performance compression accessible for modern data-intensive applications without sacrificing throughput or resource efficiency.
- OpenZL models compression as a DAG of modular codecs, enabling rapid composition of application-specific compressors.
- Beats state-of-the-art general-purpose compressors on both ratio and speed; competitive with deep-learning compressors but orders of magnitude faster.
- Internal Meta deployments reduced development timelines from months to days with consistent size and speed improvements.
Why It Matters
OpenZL enables faster, smaller data compression at scale, cutting development time and making specialized compressors practical for production systems.