How do you test AI agents in production? The unpredictability is overwhelming.[D]

A veteran QA engineer with nearly a decade of experience has hit a wall testing LLM-based agents in production. Their team's agent handles multi-step tasks, but the fundamental unpredictability of the system breaks every testing method they know. At temperature=0, the same input can produce different reasoning chains and tool selections, making assertions like 'given input X, assert output Y' impossible. Snapshot testing on final outputs is too brittle—a correctly worded but different phrasing breaks the test. Regex or keyword matching misses reasoning errors that accidentally land on the correct answer. Human evaluation doesn't scale and isn't automatable. Even evals with scoring rubrics lack clear pass/fail thresholds.

The post's author wants a framework equivalent to integration tests for reasoning steps: verifying that given a tool result, the next step correctly incorporates it. But making that assertion without hardcoding expected outputs or using another LLM as a judge introduces new failure modes. The agent runs inside the product with real consequences when it makes a bad call. This highlights a growing crisis in AI quality assurance as agents move from prototypes to production systems, where traditional QA instincts don't map and no established framework exists for testing agentic reasoning at scale.