FRAME lets LLMs learn optimal fine-tuning domain per task

Researchers from an academic team have introduced FRAME (Fractional-Fourier Mixture of Experts), a novel parameter-efficient fine-tuning (PEFT) method that treats the choice of adaptation domain as a learnable parameter. Traditional PEFT methods work either in the spatial domain (like LoRA) or the Fourier domain (spectral adapters). FRAME bridges these by assigning each expert a single learnable scalar: its fractional Fourier order. This order continuously interpolates between the spatial and spectral domains, allowing the model to place each low-rank update in the domain where it is most compact. Because fractional-Fourier operators of different orders are mutually incoherent, experts naturally decorrelate, reducing interference and improving multi-task composition.

FRAME was evaluated on LLaMA-3.1-8B and Qwen2.5-7B across commonsense reasoning, math, code, and knowledge benchmarks. It consistently outperforms strong baselines—including FlyLoRA, FourierMoE, and HMoRA—while keeping a small active-parameter budget. The fractional order is trained with a separate optimizer, and the transform uses a chirp-FFT surrogate with O(d log d) complexity, adding negligible cost over standard MoE-LoRA. Analysis shows that learned orders specialize interpretably: low-level layers favor spatial domains, while high-level layers shift toward spectral domains. This work demonstrates that learning the adaptation domain itself yields more efficient and composable fine-tuning.