[R] Prompt Repetition Shows Null Result on Agentic Engineering Tasks (n=20, blind scored)

A new research study reveals surprising efficiency gains when using prompt repetition with AI agents. Researchers tested Claude Haiku 4.5 agents on engineering tasks in a blind-scored, pre-registered experiment with 20 trials. While both control and treatment groups achieved perfect 100% scores on task completion, the agents receiving repeated prompts demonstrated significantly better operational efficiency.

The technical findings show treatment agents completed tasks in fewer conversational turns and used 13% fewer output tokens compared to control agents. This efficiency improvement occurred despite identical final outcomes, highlighting a limitation in current evaluation methods. Standard fixed-format benchmarks typically measure only final accuracy, missing these important efficiency metrics that could translate to real-world cost savings and faster task completion.

This research matters because it challenges how we evaluate AI agent performance. As companies increasingly deploy agents for engineering and development tasks, understanding efficiency metrics beyond simple accuracy becomes crucial. The 13% token reduction represents potential cost savings in production environments, while fewer turns could mean faster task completion in real workflows. Though the study acknowledges limitations including small sample size and potential confounding factors, it points toward more nuanced evaluation frameworks for agentic AI systems.