LangSmith on AWS offers frameworks for evaluating AI agent reliability

Evaluating AI agents is notoriously difficult due to their non-deterministic, multi-step nature where a single bad tool call can cascade through an entire workflow. LangSmith on AWS offers a comprehensive evaluation framework to catch these issues early, track them in production, and continuously improve agent reliability. The post combines learnings from LangChain and Anthropic into a practical guide covering five evaluation patterns for deep agents: benchmarking at the task level, running multiple trials per task to account for non-determinism (using pass@k and pass^k metrics), grading along multiple dimensions (trajectory, final response, and other state), building offline evaluations with pytest and LangSmith, and configuring online monitoring for production.

The walkthrough uses a text-to-SQL agent powered by Amazon Nova 2 Lite, a fast, cost-effective reasoning model available in Amazon Bedrock. Nova 2 Lite supports extended thinking with configurable budget levels (low, medium, high) and accepts text, image, video, and document inputs with a 1 million-token context window. It excels at instruction following, function calling, and code generation—critical for agentic workloads. The article explains key evaluation terminology (tasks, trials, graders, transcripts, outcomes) and highlights the three properties that make agent evaluation harder: non-determinism, error propagation, and creative solutions. By adopting these patterns, teams can test trajectory (tool call sequences), final responses, and other artifacts, ensuring agents are reliable before deployment.