New AI Benchmark Could Improve Farsi Chatbots for 120 Million People

Researchers just built a new benchmark called TalkFa to train AI chatbots and voice assistants in Farsi (also known as Persian), the language spoken by over 120 million people worldwide. Think of TalkFa as a giant practice gym where AI systems learn to understand and respond naturally in everyday conversations.

The team created three different training datasets: one with Wikipedia-based conversations, another with annotated dialogue acts and emotions, and a third with theatrical dialogues. Every single conversation was reviewed and improved by native Farsi speakers before being used for training. This ensures the AI learns real, natural ways people actually talk, not just robotic textbook phrases.

In testing, the new benchmark helped AI models learn faster and perform better, even using less training data. But here's the important catch: when researchers compared AI performance with automatic metrics (computer scoring), they found these scores were overly optimistic compared to real human evaluations. This means AI systems might seem better than they really are without human testing.

The researchers will release all the training data, guidelines, and code for free, giving companies and developers the tools to build better Farsi AI tools. Early testing shows this remains a tough challenge, with even advanced AI systems struggling to match human-level conversation quality.