Amazon SageMaker AI's new observability monitors both GPU usage and LLM quality

Deploying large language models (LLMs) at scale on Amazon SageMaker AI requires observability that goes beyond traditional software metrics. Unlike deterministic applications, LLMs generate variable responses whose quality can degrade over time due to input distribution shifts. This new solution addresses two complementary dimensions: quantity (infrastructure health) and quality (LLM output performance). Quantity monitoring tracks request throughput, GPU/CPU utilization, latency, and error rates per model using enhanced metrics automatically published by SageMaker AI to CloudWatch. Quality monitoring captures composite scores for accuracy, safety, and consistency, published to a separate CloudWatch namespace to keep signals cleanly separated.

At the core of the architecture are inference components, letting teams host multiple LLMs (e.g., gpt-oss-20b and Qwen2.5-7B-Instruct) on shared infrastructure while maintaining per-model isolation for traffic routing and scaling. Amazon Managed Grafana consolidates both metric streams into unified dashboards, enabling correlation between infrastructure health and output quality. For example, an endpoint may appear operationally healthy while producing unsafe responses, or deliver high-quality outputs while running inefficiently on over-provisioned instances. The solution also supports thresholds, automated alerts, and comparative analysis across models to continuously tune cost, performance, and quality.