AI Hobbyist Makes Chatbots 60% Cheaper on a $300 Graphics Card

A hobbyist with a $300 graphics card (RTX 4060) has successfully converted two small AI language models into a more efficient type called Mixture-of-Experts (MoE). Normally, big tech companies spend millions to do this, but this experiment shows it's possible on a budget. The converted models use only about 40-44% of the computing power per word compared to the originals, which could lead to cheaper AI services.

However, there's a catch: the converted models are less accurate. For example, on a test of common sense reasoning, the original model scored 46.7% correct, while the converted version scored 40.8%. On another test, the drop was from 51.4% to 41.7%. So while they're cheaper to run, they make more mistakes. The hobbyist notes that at this small scale, the performance improvement is minimal because the models are already tiny.

The technique works by splitting the model's 'brain' into multiple experts. For each word, only a few experts are used, saving computation. The hobbyist converted models with 32 experts, using the top 8 for each word, plus a shared expert. Some layers were kept dense because they were hard to convert, slightly increasing compute but keeping quality.

This experiment is mainly for learning and fun, but it points to a future where AI can run on cheaper hardware. The hobbyist plans to try larger models next, but needs more computing power. If you find this interesting, you can support them on Ko-fi. Ultimately, this could lead to more accessible AI for everyone, from small businesses to individual developers.