I tested as many of the small local and OpenRouter models I could with my own agentic text-to-SQL benchmark. Surprises ensured...

Developer Nick Lothian has launched a new, publicly accessible benchmark designed to stress-test AI models on a critical real-world task: agentic text-to-SQL generation. The benchmark presents models with 25 complex English queries—like calculating revenue per unit and margin percentages across product subcategories—and challenges them to produce executable SQL. Crucially, the AI agent can see query results and engage in limited debugging rounds to fix errors, mimicking a practical development workflow. The test is deliberately short and fast, running in under five minutes for most models, but is "tough enough to separate the best models from the others." A key innovation is the integration of a Llama.cpp WASM version, allowing anyone to run the benchmark against their own local server or via OpenRouter.

The results have yielded significant surprises, reshuffling expectations around model capabilities. Leading the pack among open models are Kimi-K2.5 and two Qwen variants: the massive 397B-A17B and the surprisingly efficient 27B parameter model. In a notable upset, NVIDIA's Nemotron-Cascade-2-30B-A3B outscores the larger Qwen 3.5-35B-A3B and matches the performance of Codex 5.3. Lothian also highlights Mimo v2 Flash as an underrated "gem." The benchmark is now live for community validation, with Lothian actively soliciting scores and feedback for a planned v2, aiming to create a standard for evaluating practical, agentic reasoning in data tasks.