Stanford researchers unveil LARA for dynamic AI adaptation

Researchers from Stanford have unveiled LARA (Lightweight Additive Residual Adaptation), a novel approach to AI model adaptation that operates in the residual stream rather than modifying base model weights. Unlike traditional methods like LoRA (Low-Rank Adaptation), which inject low-rank updates into weight matrices, LARA reads hidden states at specific layers and applies low-rank corrections directly to the residual stream. This preserves the original model's weights while enabling efficient adaptation.

The breakthrough enables a single frozen 1.5B parameter model to host seven distinct behaviors—including six fine-tuned models and one preference-optimized model—with just 33MB of additional overhead. This represents a 99% reduction in memory compared to storing separate models for each behavior. The system allows behaviors to be routed automatically per token, and a scaling parameter (γ) enables smooth interpolation between base and adapted behaviors. The approach supports hosting many behaviors simultaneously, making it ideal for multi-functional AI systems on edge devices.