RSLoRA slashes LoRA tuning costs with training-free rank allocation

RSLoRA (Representational Sensitivity LoRA) tackles a core inefficiency in parameter-efficient fine-tuning: the conventional practice of assigning the same rank to all LoRA modules ignores that different neural layers have very different functional roles. Existing rank allocation methods fall into two camps—training-based methods that incur prohibitive overhead, and pre-allocation heuristics that fail to capture task-specific representation manifolds. RSLoRA sidesteps both by introducing a virtual representational probing mechanism that is entirely training-free and gradient-free. The key insight is a "sensitivity regime shift" across layers: static weight analysis and local gradients do not reflect how updates reshape internal representations. By injecting structured low-rank noise into each module and measuring the resulting manifold displacement using Effective Rank and Fréchet Distance, RSLoRA pinpoints high-sensitivity modules that need higher rank capacity—all without a single backward pass.

Extensive evaluations on mainstream computer vision benchmarks show RSLoRA consistently outperforms state-of-the-art allocators like AdaLoRA and GoRA. The method eliminates iterative training-time adjustments and backward gradient computations, making it highly efficient and robust for large-scale model adaptation. For practitioners fine-tuning massive vision models, RSLoRA offers a plug-and-play rank allocation strategy that respects representational impact without adding computational overhead. The paper includes 16 pages of analysis and 3 figures, and is available on arXiv (2607.09757). This represents a practical step toward more intelligent, geometry-aware fine-tuning that scales gracefully.