Heretic 1.3 released: Reproducible models, integrated benchmarking system, reduced peak VRAM usage, broader model support, and more
Reproducible model abliteration? Heretic 1.3 ensures byte-identical results across machines.
Deep Dive
The open-source project Heretic (20k GitHub stars, 13M downloads) releases version 1.3 with reproducible runs, enabling byte-for-byte identical model outputs via an optional reproduce directory. It also adds an integrated benchmarking system (MMLU, EQ-Bench, GSM8K, HellaSwag) that runs directly in Heretic without exporting, making it easier to decide whether a model is worth publishing.
Key Points
- Reproducible runs ensure byte-for-byte identical model outputs via a complete reproduce directory on Hugging Face.
- Integrated benchmarking runs MMLU, GSM8K, EQ-Bench, and HellaSwag directly in Heretic without extra configuration.
- Version 1.3 also reduces peak VRAM usage, broadens model support, and builds on 20k GitHub stars and 13M downloads.
Why It Matters
Ensures trust and reproducibility in model decensoring, letting AI researchers verify results and compare quality confidently.