Backtrader-Bench: Tool-augmented LLM agents hit 90% accuracy in trading

Backtrader-Bench, a new benchmark from researchers Ruoxi Zhao and Maziar Raissi, tackles a thorny problem: how to fairly evaluate LLM coding agents in algorithmic trading. Static benchmarks risk data contamination, and numerical backtest outputs require ground-truth from actual code execution. The framework instead uses two complementary pipelines. First, a deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations spanning five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer. Second, a generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution.

On a curated 30-question set, the benchmark evaluated 11 models without tools (10 runs each) and four with-tools configurations. Tool-augmented agents reached 90.0% accuracy in a single pass—GPT-5.5 and Opus 4.7 led the pack—outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tool accuracy dropped sharply, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows. Accepted to the FinLLM Workshop at IJCAI 2026, This benchmark offers a contamination-resistant way to measure and improve LLM-driven trading systems.