JAXBench: 50-Workload Benchmark for AI-Driven TPU Kernel Optimization

Researchers from Google have released JAXBench, a comprehensive benchmark suite for measuring and advancing autonomous TPU kernel optimization. The benchmark comprises 50 JAX workloads carefully selected for relevance and optimization headroom. Seventeen of these are production machine learning operators extracted from architectures in Google's open-source MaxText library, including Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, and AlphaFold2. Eight of these production operators ship with hand-optimized Pallas kernels from the Tokamax library, establishing an expert performance upper bound. The remaining 33 operators are ported from the GPU-focused KernelBench suite, with corrected implementations and problem sizes re-tuned to achieve high MXU utilization on Google's TPU v6e hardware.

To evaluate progress, the team tested four feedback-driven methods for generating candidate Pallas kernels. The results highlight that for a sparsely-documented domain-specific language like Pallas, providing target-specific TPU documentation far outweighs scaling up model size. Conditioning Gemini 3 Flash on curated TPU documentation raised the per-sample correctness rate from 5.8% to 37.3%, enabling the system to solve 48 of the 50 benchmarks at an average 1.28x speedup over XLA. Further, search-based methods like Autocomp's beam-search pipeline pushed performance to a 1.36x geomean speedup across the full suite. On the eight hand-tuned kernels, Autocomp achieved 1.60x over XLA, recovering most of the expert 2.08x Tokamax upper bound, though it still trails on specialized paged and ragged attention operators. The authors release JAXBench along with evaluation harness and baseline results to foster open-source contributions.