APEX4 achieves up to 2.09x speedup on W4A4 LLM inference via compute rebalancing
Outperforms FP16 by 78% on RTX 3090 while keeping accuracy within 0.63 perplexity
Researchers from Hasso Plattner Institute and other institutions have introduced APEX4, a system for pure W4A4 (4-bit weights and activations) LLM inference that tackles the longstanding bottleneck of group dequantization overhead on CUDA Cores. Through extensive benchmarks across Ampere and Ada GPUs, they identified the Tensor Cores to CUDA Cores throughput ratio (ρ) as the key hardware indicator. On RTX 3090 (ρ=16) and A40 (ρ=16), the W4A4-g128 kernel achieves 2.0–2.5× compute speedup over FP16, while on A100 (ρ=64) it degrades to 0.43–0.47×. This shows W4A4 viability is platform-dependent.
APEX4 co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation, mitigating the CUDA Cores dequantization bottleneck. When deployed as a drop-in replacement in unmodified vLLM, APEX4 delivers up to 1.66× end-to-end speedup on L40S (ρ=8), 1.78× on RTX 3090 (ρ=16), and 2.09× on A40 (ρ=16). For A100 (ρ=64), a mixed-granularity mode recovers 1.20–1.40× speedup. Accuracy stays within 0.63 perplexity of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0–4.4% in zero-shot accuracy, offering practical deployment gains.
- APEX4 uses ρ (Tensor Core to CUDA Core throughput ratio) to dynamically adjust quantization granularity, solving the dequantization bottleneck on A100 while benefiting consumer GPUs like RTX 3090.
- Achieves 1.78× end-to-end speedup on RTX 3090 and 2.09× on A40 over FP16 in unmodified vLLM, with perplexity within 0.63 of FP16.
- Outperforms W4Ax Atom-g128 by 4.0–4.4% in zero-shot accuracy on LLaMA-2-70B, demonstrating both speed and accuracy improvements.
Why It Matters
APEX4 makes 4-bit LLM inference practical on consumer GPUs, enabling faster deployment of large models without accuracy loss.