Research & Papers

Kubernetes Evidence Horizon Taxonomy Formalizes Diagnostic Data Loss

Five distinct boundaries destroy critical pod lifecycle context within minutes

Deep Dive

Kubernetes clusters generate rich operational events during pod lifecycle transitions, but the platform's native event retention model systematically discards the most diagnostically valuable context. Shamsher Khan's paper formalizes these loss mechanisms as an Evidence Horizon Taxonomy with five distinct boundaries: H1 (LastTerminationState rotation, ~90s) destroys container failure forensics; H2 (scheduler event pruning, 1hr/1000-event cluster limit) destroys placement rationale; H3 (ephemeral container exit, immediate) destroys debug session context; H4 (kubelet reconciliation gap) destroys in-memory operational state; and H5 (scrape-interval blind spot) renders sub-interval pod lifetimes invisible to poll-based observability.

The paper extends the Operational Memory Architecture (OMA) to address the full taxonomy. Two new causal patterns are defined: P004 (Scheduler Decision Provenance) captures FailedScheduling predicate failures before kube-apiserver TTL pruning and demonstrates the first cross-horizon causal chain linking scheduler evidence to downstream OOMKill failures. Two new Go watchers (EventWatcher, EphemeralWatcher) and two new storage tables (scheduler_events, ephemeral_exits) extend the original architecture. Validated on Minikube (3-node) and AKS 1.32.10. The original 30-run statistical latency analysis (242 edges, intra-cycle mean 0.702ms) and stress evaluation (2.86 events/sec at 20 pods, 8.8MB RAM) are carried forward and augmented with H2, H3, and H5 results. Implementation available on GitHub.

Key Points
  • Five Evidence Horizon boundaries (H1-H5) define where Kubernetes permanently discards diagnostic context, with H1 destroying container failure forensics in ~90 seconds.
  • Two new causal patterns (P004, P005) capture scheduler decision provenance and ephemeral debug context, enabling cross-horizon troubleshooting.
  • Extended OMA adds EventWatcher and EphemeralWatcher Go watchers with dedicated storage tables, achieving 0.702ms mean latency at 2.86 events/sec.

Why It Matters

For SREs and DevOps engineers, this taxonomy reveals blind spots in Kubernetes observability and offers a practical patch for forensics.

📬 Get the top 10 AI stories daily