Microsoft researchers release SentinelBench for long-running AI agents

Microsoft Research has released SentinelBench, a new open-source benchmark designed to evaluate AI agents on long-running monitoring tasks. The benchmark addresses a critical flaw in current agent evaluation: most benchmarks assume agents should constantly take actions (tool calls, page refreshes, searches), but many real-world tasks—like waiting for an email reply or a calendar invite—require sustained attention rather than constant activity. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment features a live web interface that replays a scripted sequence of events, forcing agents to navigate and reason about web pages whose state changes over time.

The benchmark measures three key metrics: task completion, reaction time (how quickly the agent responds to an event), and resource use (e.g., API calls, compute). This exposes the fundamental tradeoff between responsiveness and cost—an agent that polls constantly may react quickly but wastes resources, while one that waits too long may miss opportunities. The research team tested three models and two browser-agent harnesses, establishing baseline performance and showing how different agent design choices dramatically impact results. The paper demonstrates that SentinelBench can distinguish meaningful differences in agent behavior, making it a valuable tool for developers building production monitoring agents that need to balance efficiency with timeliness.