Amazon Built an AI Babysitter to Watch Its Other AIs

AI agents — software that can book flights, issue refunds, or answer customers on its own — fail in ways ordinary software doesn't. A broken website throws an error message. An AI agent might quietly send you to the wrong department, or simply stop completing reservations while every dashboard stays green. Traditional monitoring checks whether the software ran, not whether the AI actually helped the person using it.

Amazon just released two tools to close that gap. AgentCore Evaluations acts like a constant quality grader, scoring every live AI conversation on helpfulness, correctness, and whether the user's goal got accomplished. AWS DevOps Agent plays the role of an automated on-call engineer — when something breaks, it traces the failure across systems, reads the logs, and pinpoints the root cause without a human staying up all night.

To prove it works, Amazon built a demo airline reservation system staffed by four specialized AI agents under one supervisor. The scariest failures turned out to be silent ones: a permission gets quietly revoked, a booking bot stops reserving, yet the logs still show successful steps because the problem happened three calls deep in a chain. No error code, no alarm — just confused customers and lost revenue.

The catch: these tools only work inside Amazon's cloud, take real engineering effort to set up, and rely on AI to judge other AI — which is not foolproof. Still, as companies hand more genuine work to AI "employees," having something watch the watchers stops being a nice-to-have and starts looking mandatory.