Intel AI PC fleets run 70B LLMs via distributed pipeline shards

A new arXiv preprint (2608.19147) from Intel researchers describes a method to turn fleets of idle Intel AI PCs into distributed LLM inference servers. Modern AI PCs ship integrated GPUs, NPUs, and 16+ GB of unified memory, but that's insufficient for large models like a 70B-parameter LLM. The authors demonstrate that a handful of these machines, connected over an ordinary network, can serve models beyond any single PC's capability by employing pipeline parallelism: the model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so every machine runs one shard and passes activations to the next.

Three techniques make this practical. First, they recover performance by injecting a beam_idx Gather into each shard, which triggers OpenVINO's IndirectKVCache fusion and brings shards to parity with monolithic inference. Second, they leverage speculative decoding on stateful OpenVINO models. Third, they interleave multiple users' requests across stages using micro-batching, with each request carrying its own cache. Results show a two-node Llama 3.1 8B INT4 pipeline serving two concurrent users at 1.79x the single-user throughput of the unsplit model on identical hardware, with the gap widening under simulated wide-area latency. The same design scales to a 70B model on four Lunar Lake AI PCs running on Intel Tiber Cloud, delivering interactive-speed single-user output that is token-for-token identical to non-speculative decoding. Code and reproduction scripts ship as a self-contained package.