LLT Transformer slashes PDE training time by 2.5x

Neural operators have become a popular approach for learning solution maps of partial differential equations (PDEs), but standard attention mechanisms in transformer-based operators scale quadratically with the number of computational nodes and lack an explicit bias toward local interactions. The newly proposed Local Linear Transformer (LLT) from Ovadia and Turkel addresses both issues by combining linear global attention with local spatial mixing, and incorporating coordinate and geometry information. This design allows LLT to capture long-range dependencies efficiently while maintaining strong local accuracy.

LLT was evaluated on a diverse set of PDE problems—elasticity, plasticity, airfoil flow, pipe flow, Darcy flow, and a 3D car aerodynamics dataset with 32,186 unstructured mesh points per sample—spanning finite-element, finite-volume, and finite-difference discretizations on both structured and unstructured meshes. Compared to existing neural operator and transformer baselines, LLT achieves competitive or lower relative L2 error while reducing wall-clock time per training iteration by factors of 1.8 to 2.5 relative to Transolver on matched structured discretizations. These results demonstrate that LLT provides an accurate and computationally efficient operator that generalizes across discretizations, mesh types, and problem settings.