Pyrecall open-source tool detects catastrophic forgetting during LLM fine-tuning
Catastrophic forgetting is a known risk; Pyrecall offers a free, local fix.
Catastrophic forgetting remains a critical challenge when fine-tuning large language models (LLMs) — training on new tasks often erodes previously learned capabilities. While research on continual learning is extensive, practical tooling to detect and prevent this has been lacking. Enter Pyrecall, a new open-source Python library developed by Reddit user Level_Frosting_7950. The tool snapshots skill scores before fine-tuning, then compares them after training to flag any regressions. When forgetting is detected, Pyrecall can roll back specific LoRA adapters by name, preserving prior competencies.
Pyrecall is fully local (no external API calls), distributed under the MIT license, and installable via pip (v0.1.0). The developer admits uncertainty about the benchmark design and invites community feedback. For teams fine-tuning LLMs on sensitive or proprietary data without cloud dependence, Pyrecall offers a lightweight, auditable safety net. Its approach mirrors popular continual learning strategies but packages them into a simple CLI and Python interface.
- Snapshots skill scores before/after fine-tuning to detect regressions in LLM capabilities.
- Rolls back LoRA adapters by name when forgetting is detected, preserving prior performance.
- Fully local, MIT licensed, and installable via pip install pyrecall — no external APIs required.
Why It Matters
Enables fine-tuners to safely iterate without degrading model quality, using a free, local tool.