[P] Bypassing CoreML to natively train a 110M Transformer on the Apple Neural Engine (Orion)

A developer has open-sourced ORION, a groundbreaking system that bypasses Apple's restrictive CoreML framework to enable direct programming and on-device training of large language models on the Apple Neural Engine (ANE). This work, building on foundational hardware reverse-engineering by 'maderix', addresses the core frustration in Apple's ML stack: CoreML's opaque abstractions prevent direct ANE access and do not support training, leaving the chip's significant compute—up to 38 TOPS (INT8) and ~19 TFLOPS of fp16—largely untapped for LLMs. ORION represents a bridge from raw hardware exploit to a mathematically stable runtime, developed through an 'architectural delegation' approach using Claude AI to generate low-level code.

The technical achievement is substantial, requiring the cataloging of 17 undocumented ANE programming constraints (11 newly discovered) and solving the 'numerical stability ceiling' that caused previous attempts to diverge with 100% NaN rates. ORION's custom compiler lowers a 27-operation graph IR through five optimization passes to emit ANE-native MIL. Key fixes included a deferred compilation pipeline to prevent stale programs, activation clamping to handle fp16 overflow, and strict gradient sanitization. The result is stable multi-step training of a 110M-parameter model and inference performance of 170+ tokens/s for GPT-2 124M on an M4 Max, demonstrating the real leverage of directly programming Apple's specialized silicon.