Thoughts on using an AMD Alveo V80 FPGA PCI card as a poor man’s Taalas HC1 (LLM-burned-onto-a-chip).

A developer has revived the concept of FPGA-based AI inference, proposing the AMD Alveo V80 PCI card as a cost-effective alternative to the upcoming Taalas HC1, which burns LLM weights directly onto a chip for speeds of 15,000 tokens per second. Drawing from crypto mining experience with ASICs and FPGAs, they ran the idea by Gemini Pro for a feasibility check. The Alveo V80, with 16GB of HBM and a $9,500 price tag, was found capable of using speculative decoding to accelerate inference. Gemini estimated speeds of 3,200 tk/s for a Q4 quantized Qwen3.5 4B model and 1,400 tk/s for the 9B version. While not matching Taalas's raw speed, the FPGA offers reprogrammability, avoiding the weight lock-in of ASIC-style chips.

This approach revives the FPGA's promise from the crypto era: better speed than GPUs for specific tasks without becoming obsolete when algorithms change. The developer notes that the Alveo V80's HBM can store model weights, allowing the FPGA to act as a programmable inference accelerator. This could democratize high-speed LLM serving for smaller teams or research labs that can't afford custom ASICs. However, the setup requires significant engineering effort, including custom firmware and quantization pipelines, and the 16GB memory limits model size. The post invites community feedback on whether anyone has attempted similar FPGA-based inference, hinting at a niche but potentially powerful use case for repurposed hardware in the AI boom.