New Test Reveals AI Design Agents Make Costly Mistakes

Imagine hiring an AI to design your company's new logo or packaging. It's fast and cheap, but what if it ignores your brand colors or forgets to include a print-ready file? That's the scenario a new study, DRAFT-Bench, explores. Researchers tested two leading AI design agents—OpenAI's Codex and Anthropic's Claude—on 35 realistic design projects, from brainstorming to final delivery. The AI completed most tasks, but when checked against the original instructions, it made mistakes in about a third of cases for Codex and a sixth for Claude.

The test simulated the full design process: understanding client needs, exploring creative options, making revisions, and preparing files for production. While both AIs handled the steps, they sometimes missed key requirements. For example, in one task, the AI might use the wrong font or forget to save the file in the right format. These errors could cost businesses time and money if they rely on AI without human oversight.

The study also looked at a separate task: responding to a request for quote (RFQ). Both AIs did well, but the results varied depending on the scenario, making it hard to declare a clear winner. The researchers caution that their test is small and uses model judgments rather than expert reviews, so it's not a final verdict on AI's design abilities. Still, it highlights that AI can handle the mechanics of design but may struggle with the nuances of professional standards.

For everyday people, this means AI design tools are promising but not yet reliable for critical work. They can save time on routine tasks, but you'll still need a human designer to catch mistakes. As these tools improve, they could become valuable assistants, but for now, don't expect them to replace professionals entirely.