DeepSeek V3.2 agent harness hits 67% on ARC-AGI-1
Two-stage pipeline and reflective orchestrator lift baseline by 52 points without fine-tuning.
A new paper by Kabir Moghe and Peter Chin demonstrates that open-weight models in non-thinking mode can achieve state-of-the-art abstract reasoning on ARC-AGI-1 through clever architecture alone. The authors introduce two agentic harnesses built on DeepSeek V3.2: an Explorer-Definer Pipeline that separates pattern discovery from program synthesis, and a Reflective Orchestrator that autonomously explores new transformations when initial hypotheses fail. The pipeline achieves 57.5% pass@2 at $0.25 per task, and the orchestrator pushes that to 67.25% at $0.62 per task—a ~52-point lift over the 15.5% one-shot baseline.
The study also reveals that the pipeline is generation-bound rather than selection-bound: unbiased pass@k analysis shows that selection via training-pair accuracy captures ~95% of the candidate ceiling, meaning future gains require broader generation, not better ranking. The orchestrator confirms this by delivering a +9.81 pp unbiased pass@1 lift. An ablation identifies the 'think tool' as critical—removing it drops pass@2 by 5.75 pp. This work offers a cost-effective alternative to heavy compute or benchmark-specific training, making advanced reasoning more accessible.
- Explorer-Definer Pipeline achieves 57.5% pass@2 at $0.25 per task on ARC-AGI-1 public set
- Reflective Orchestrator reaches 67.25% pass@2 at $0.62 per task, lifting baseline by ~52 points
- Pipeline is generation-bound; removing 'think tool' reduces pass@2 by 5.75 pp
Why It Matters
Proves open-weight models with smart architectures can rival heavy compute methods on abstract reasoning at a fraction of the cost.