AWS Trainium Frontier competition offers $25K to train LLMs on purpose-built chips

AWS's Annapurna Labs is funding a new competition, Trainium Frontier, to empirically explore how language model architectures should change when trained on purpose-built accelerators rather than conventional GPUs. The premise is that modern LLM designs—attention patterns, MLP structures, numerical formats, parallelism strategies—have been shaped entirely by Nvidia-style hardware constraints like warp sizes and tensor core geometries. Trainium offers a different design surface: more on-chip SRAM (SBUF), explicit software control over data movement, energy-efficient systolic matrix multiplication, and a memory hierarchy built for training and inference flows. This shifts the TFLOPs-to-bandwidth ratio, potentially making compute-bound operations that were memory-bound elsewhere, opening room for architectures that trade extra compute for reduced memory traffic.

The competition kicks off with a 50M-parameter nanochat-derived baseline (a GPT-style dense LLM with RMSNorm, rotary embeddings, and ReLU² MLP) that participants can modify entirely—architecture, optimizer, training loop, and even custom NKI kernels. Phase 1 gives each team 30 minutes on a single Trainium2 chip, optimizing validation bits-per-byte. The top 10 then move to Phase 2: 4 hours on a full Trainium server, where inference performance is also scored across downstream in-context learning tasks. The fixed time budget creates a direct tradeoff: better architectures reduce steps needed, while faster kernels squeeze more steps into the same window. Winners present at NeurIPS 2026 in Sydney, with $25,000 first place, $10,000 second, $5,000 third, co-publication with Annapurna researchers, and exclusive swag for the top 10. The goal is to discover what optimal LLMs look like when hardware assumptions change—research that could inform future AI chips and architectures.