IBM's Granite-Switch architecture revolutionizes LoRA with CPU support

IBM has introduced Granite-Switch, a groundbreaking architecture for its Granite-4.1 model that embeds multiple LoRA (low-rank adaptation) adapters directly into the model, allowing for per-token adapter selection using control tokens. This innovation enables dynamic switching between specialized adapters mid-sequence without requiring GPU acceleration, making it viable for CPU-only environments. The architecture was demonstrated end-to-end on a 3B parameter checkpoint, with performance verified on both base and control-token paths.

The implementation includes a novel in-graph router mechanism that replaces a previous global sticky index, addressing critical issues like concurrency and multi-turn isolation. The router uses a single-head causal attention layer to recover adapter indices per token, ensuring thread-safe and sequence-isolated adapter switching. The team also released self-contained scripts for Apple Silicon (Metal) builds and mid-sequence adapter-switch demos, showcasing practical use cases like answerability toggling and query rewriting. This approach mirrors mechanisms in vLLM and Hugging Face backends but adapts them for CPU and on-device inference scenarios.