FBLayout: New framework speeds up LLM finetuning on mobile GPUs by 5.7x

On-device finetuning of large language models (LLMs) promises privacy-preserving personalization but is hindered by mobile GPU memory constraints and inefficient layout transformations during attention mechanisms. Existing frameworks either use unified layouts that fragment memory during backpropagation or rely on expensive explicit conversions. FBLayout tackles this with three key innovations: (1) a unified R-Tile layout for multi-dimensional reductions across both forward and backward passes, (2) tile-based index transformation that eliminates physical data movement, and (3) activation-guided layout selection to propagate efficient layouts globally across the computation graph.

Evaluated on seven transformer models (including GPT-2, BERT, and Llama variants) using ARM Mali and Qualcomm Adreno GPUs, FBLayout delivers 2.2-5.7x end-to-end speedup over MNN, TFLite, and TVM. It also significantly improves L1/L2 cache hit rates and reduces peak memory usage, making large-model fine-tuning practical on phones. The paper will appear at Mobisys 2026 and is available on arXiv.