PermDoRA study shows adapter interference in LLMs isn't geometry-driven

A new paper from Sivaramakrishnan et al., PermDoRA, challenges a widely held hypothesis in modular LLM adaptation. The common belief is that interference when composing multiple adapters (e.g., for different domains) arises from overlap in linear parameter updates, implying that enforcing orthogonality or directional independence should improve performance. The team tested this using DoRA-RBAC, a hierarchical framework based on weight-decomposed low-rank adaptation, on two popular models: LLaMA-3.1-8B and Mistral-7B. They compared conventional Euclidean merging (simple averaging) with a geometry-aware Riemannian strategy that approximates the Fréchet mean via normalized directional averaging across four QA benchmarks: GPQA, PubMedQA, SimpleQA, and WMDP.

The results are striking: while single-domain performance on each benchmark matched standard LoRA, the geometry-aware merging provided no consistent advantage over standard averaging in multi-domain setups. Further analysis revealed that angular alignment and orthogonality of adapter updates are weak predictors of composition performance. The authors conclude that adapter interference is not governed primarily by parameter-space geometry, but instead is consistent with interactions in shared nonlinear representations. This suggests that future efforts to improve modular adapter composition should focus on understanding and mitigating interference at the representation level rather than relying on geometric constraints in parameter space.