DeepSeek's New AI Matches Top Models, But Scores Depend on Testing
AI test scores can be misleading—here's what you need to know.
DeepSeek, a Chinese AI company, has released a new model called DeepSeek-V4.1-Flash-1017. On its own model card, it claims a score of 42.3% on Terminal-Bench 4.0, a test that measures how well AI can perform complex tasks in a command-line environment. That would put it on par with Fable 5, a leading model released in June 2026. But independent evaluators like AA and Vals Index report much lower scores: 34% and 31.82%, respectively.
The discrepancy likely comes from different testing setups. DeepSeek used the Claude Code harness to run the benchmark, while AA and Vals Index probably used a different tool called mini-swe-agent. Also, each task has an 8-hour time limit, so the speed of the hardware and the AI provider's token throughput can affect results. This means the same model can score differently depending on how it's tested.
For everyday users, this matters because AI companies often tout impressive benchmark numbers to attract customers and investors. But if those numbers aren't comparable across different test conditions, they can be misleading. When you see a claim that an AI is 'as good as' another, ask: under what conditions? The real-world performance you experience may vary.
Ultimately, benchmarks are a useful but imperfect guide. They can help you compare models, but they shouldn't be the only factor in your decision. Look for independent evaluations and consider your own use case. As AI models proliferate, transparency in testing will become even more important.
- DeepSeek's new model claims top-tier performance, but independent tests show lower scores.
- Different testing tools and time limits can cause score variations.
- Always check how AI benchmarks are conducted before trusting them.
Why It Matters
AI benchmark scores can be misleading, so don't rely solely on them when choosing AI tools.