AWS launches Nova Sonic Test Harness for automated voice agent evaluation

AWS has open-sourced the Nova Sonic Test Harness, a framework designed to solve the fundamental challenge of testing speech-to-speech AI agents. Unlike text-based chatbots, voice agents operate on bidirectional audio streams, produce non-deterministic responses, and require multi-turn context tracking. The harness replaces manual testing, where teams previously had to physically speak and listen through every conversation scenario—a process that could take days for 150 scenarios across multiple personas. The tool configures tests via JSON files, runs complete conversations with Amazon Nova Sonic automatically, and evaluates outputs using LLM-as-judge techniques. It also detects audio hallucinations, where the model's spoken output differs from its text transcript. The harness handles session limits (8-minute timeouts) and supports rapid prompt iteration by providing immediate feedback on changes.

The release addresses two critical pain points: accelerating prompt engineering and enabling regression testing. Teams can now define test scenarios, run them unattended, and receive detailed reports on whether tweaks improved accuracy or introduced regressions. The harness supports complex multi-turn interactions, tool usage verification, and scenario-based success criteria. By eliminating the need for human testers to interact with the system in real time, it scales evaluation from dozens to hundreds of tests per run. This is particularly valuable for production voice applications like appointment booking and customer support, where subtle errors (e.g., forgetting to confirm a booking) can impact user trust. The tool is available on GitHub and integrates with AWS services for deployment.