INT3 compression+fused metal kernels [R]
A solo founder's tool shrinks models to 3-bit precision with minimal accuracy loss, featuring a 2-bit KV cache for long tasks.
ReinforceAI, a project by a solo researcher and founder, has released Spiral, an advanced model compression framework. The core innovation is applying INT3 (3-bit) quantization to large language models, achieving a remarkably low accuracy loss of just +0.14 nats in perplexity—a key metric for model performance. For handling long-context or 'long-horizon' tasks, which are memory-intensive, Spiral implements an ultra-efficient 2-bit Key-Value (KV) cache. This drastically reduces the memory footprint compared to standard 16-bit or 8-bit caches, enabling longer conversations or document analysis on devices with limited RAM.
The tool is currently optimized for Apple's ecosystem, shipping with custom fused Metal kernels specifically designed for M-series Macs to maximize on-device speed. A preview of the compressed Qwen 7B model is available for immediate testing via a simple command-line install. Looking ahead, the developer is focused on further kernel optimization and expanding support to Nvidia GPUs by building Triton kernels. The project is open to compressing other models upon request, targeting architectures within 100 billion parameters, positioning Spiral as a practical tool for deploying capable AI on consumer hardware.
- Achieves INT3 model quantization with only a +0.14 nats perplexity increase, preserving high accuracy.
- Implements a memory-efficient 2-bit KV cache specifically for long-context inference tasks.
- Ships with custom, high-performance Metal kernels for Apple Silicon Macs, with a preview of Qwen 7B available.
Why It Matters
Enables running larger, more capable AI models locally on consumer devices like MacBooks by drastically cutting memory and compute requirements.