Chinese AI chips fail at coding, squeezing Nvidia supply for high-end inference

Chinese AI companies are scrambling to optimize inference workloads as a massive surge in token usage collides with restricted access to Nvidia's high-end processors. While training models requires the most advanced chips, inference—the phase where trained models generate responses—can sometimes run on domestic hardware. However, industry insiders reveal that complex tasks such as coding demand Nvidia's capabilities, creating a bifurcated market: low-tier inference runs on Chinese chips, while high-quality token generation remains Nvidia-dependent.

Demand is exploding. China's average daily token calls surpassed 140 trillion in March 2026, a more than 1,000-fold increase from early 2024, according to the National Data Administration. As AI becomes more agentic—executing real-world tasks rather than simple Q&A—the compute squeeze intensifies. Guan Jiawei, VP at inference optimization startup Approaching.AI, notes that high-tier users are willing to pay premiums for coding performance that domestic processors cannot yet reliably deliver. "If we rely solely on domestic chips for inference, they can only handle the low-quality tier—weak demand, weak monetization," he says. Software optimization is helping stretch limited Nvidia supply, but the commercial path for domestic chips remains elusive.