Luce Spark runs 35B MoE models on 16GB GPU without speed loss
Calibrated expert offloading achieves 85% of all-GPU speed on a 3090
Luce Spark is a new technique from Luce Org that dramatically reduces GPU memory requirements for running large Mixture-of-Experts (MoE) models without the typical offload speed penalty. The trick: instead of naively splitting experts across GPU and CPU, Spark learns which experts are actually hit by your traffic and keeps those 'hot' experts on the GPU, while the long tail lives in system RAM. An A3B model routes to roughly 8 of 256 experts per token, so only a fraction needs to be resident. Spark runs one command — `dflash_server <model.gguf> --spark` — on any GGUF model from the Laguna or Qwen35moe backends, and it self-tunes over time.
The key components: (1) Calibrated placement that accumulates per-expert routing frequencies from real requests and pins the most-used set, dropping cold-hit rate from 36% to ~7%. (2) A bounded async cache with a fixed ring of spare GPU slots — on a cold-expert hit, weights are copied asynchronously from pinned host memory into a spare slot while evicting the LRU entry, costing throughput but no stall. (3) A fused graph that runs the entire token as one graph instead of 40 per-layer graphs, removing submission overhead. At full GPU residency (needs 24 GB) it runs at 119 tok/s, bit-identical to all-GPU. At 60% residency (fitting any 16 GB card) it hits ~100 tok/s (85% of full speed), compared to 66 tok/s for naive uniform offload. The project is open-source on GitHub under Apache 2.0.
- Calibrated placement reduces cold-expert hit rate from 36% to ~7% by learning routing patterns from real requests.
- Bounded async cache with a ring buffer overlaps weight transfers with compute, avoiding stalls on cold misses.
- Fused decode graph achieves ~100 tok/s at 60% residency on a 3090, vs 66 tok/s for naive offload.
Why It Matters
Democratizes large MoE models by enabling them on consumer GPUs without sacrificing performance.