Most People Can't Tell Top AI Models Apart, Study Finds

A tech enthusiast ran a two-month experiment with 13 friends to see if regular people could tell the difference between high-end and low-end AI models. He built a simple website that looked like ChatGPT but secretly switched between seven different AI models, from small ones like Gemma 2B to larger ones like Qwen 3.6 27B. Participants were told they had free access to the latest ChatGPT, but they didn't know which model they were actually using.

People started complaining only when the AI was very small (9B parameters or less). The most common gripe was that the model 'does not get it.' Everyone preferred responses from the Gemma models. Some even said other models 'think too much' and sound like a confused robot. Interestingly, many participants rarely used the high 'thinking' mode—only 51 out of 7,912 requests. Yet some still thought the AI was top-of-the-line because they could see its reasoning process, even when it wasn't actually reasoning much.

The big takeaway: For everyday tasks, most people can't tell the difference between a premium AI and a cheaper one. They only notice when quality drops significantly. So if you're paying for the most expensive AI, you might be wasting money—a mid-range or even free model could be just as good for your needs. Also, flashy features like visible 'thinking' can impress users even if the AI isn't actually smarter.

This doesn't mean all AI is equal. For complex tasks, bigger models still win. But for casual use—writing emails, answering questions, brainstorming—you probably won't notice. The experiment was small and informal, so treat it as a fun hint, not a scientific study. Still, it suggests that the AI arms race might be overkill for most of us.