New CurveFP datatype cuts LLM training costs by 7-bit inference
CurveFP achieves FP8-class performance with 7-bit inference—simpler math, lower cost.
Ye Qiao’s CurveFP introduces a family of closed-product codebooks that distribute quantized magnitudes across interleaved logarithmic curves under compact block scales, enabling exact algebraic closure for products via sign XOR and integer-index updates. The system is instantiated as CurveFP eight E4C3/E5C2 for training and CurveFP seven E3C3 for deployment. In evaluations, CurveFP seven outperformed tensor-wise FP8 perplexity on four 7B–9B models with one fewer element bit and stayed within 1.32% of native quality, while CurveFP eight reduced operand NMSE across all 36 paired forward and backward GEMM comparisons.
Across three matched 128.3M-parameter triplets, CurveFP eight completed 3B-token pretraining per seed with mean BF16-inference perplexity of 22.5366 versus 22.5407 for FP8, and incurred a lower format-induced penalty in all three seeds. In a 36-cell downstream matrix, CurveFP eight-trained checkpoints achieved lower WikiText-103 perplexity in all 12 seed-format comparisons, with mixed effects on PG-19 and task deltas. The results position CurveFP as an arithmetic co-design combining FP8-class behavior, seven-bit inference, and a substantially simpler product path—potentially lowering training and deployment costs for LLMs.
- CurveFP seven achieves FP8-class perplexity on 7B–9B models using 7-bit inference (one bit fewer than FP8)
- CurveFP eight reduces operand NMSE in all 36 GEMM comparisons and completes 3B-token pretraining with mean BF16 perplexity of 22.5366 vs FP8’s 22.5407
- The approach simplifies product formation to sign XOR and integer-index updates via rational-radix logarithmic datatypes with closed products
Why It Matters
Cuts LLM training and inference costs with simpler arithmetic and 7-bit deployment without sacrificing quality.