Motorway & AWS cut agent errors from 1-in-8 to 1-in-50
Dealers find cars 50x faster with natural language – here's how they proved it works
Motorway, a UK-based online car marketplace running daily auctions of up to 2,500 vehicles for 8,000 dealers, teamed up with AWS Prototyping and AI Customer Engineering (PACE) to deploy a conversational AI agent for dealer stock searches. The agent, built on Strands Agents SDK and Amazon Bedrock AgentCore, exposes eight tools combining 89 vehicle attributes with vector search. Dealers can now ask "Find me diesel SUVs under £25k near my dealership" instead of browsing CSV files for hours. With ~1,500 concurrent peak users, reliability was critical.
To ensure trust, the team designed an end-to-end evaluation pipeline using two phases: build-time testing with strands-agents-evals and production monitoring with Amazon Bedrock AgentCore Evaluations. The three-layer framework assesses tool usage, reasoning, and output quality. A five-stage deployment pipeline enforces quality gates – releases block when metrics fall below thresholds. Results: incorrect results dropped from 12.5% to 2%, and issue detection time collapsed from hours to minutes. The companion repository provides a deployable blueprint (AWS services, but principles are system-agnostic) including least-privilege IAM roles and parameter-store secrets.
- Incorrect agent results reduced from 1-in-8 (12.5%) to 1-in-50 (2%) through structured evaluation
- Evaluation pipeline uses Strands Agents SDK for build-time tests and Amazon Bedrock AgentCore for production monitoring
- Three-layer framework covers tool selection errors, semantic search misinterpretations, and multi-turn context drift
Why It Matters
Proves conversational AI agents can be production-grade when backed by rigorous, automated quality gates and monitoring pipelines.