AgentS4D benchmark: 68% of LLM agent runs trigger safety risks

AgentS4D, a new benchmark from researchers at multiple institutions including Jiajun Zhou, addresses a critical gap in AI safety evaluation: runtime risks during execution, not just final outputs. The benchmark builds a four-dimensional framework covering six risk-entry sources, six induction strategies, nine target harms, and seven lifecycle checkpoints. It contains 328 risk-injected cases that simulate real-world workspace scenarios where agents interact with files, APIs, and persistent state.

The researchers evaluated 20 agent configurations—four harnesses (Hermes, OpenClaw, Claude Code, Codex) paired with five LLM backends (GPT-5.5, Gemini 3.1 Pro, DeepSeek-V4-Pro, MiniMax-M3, Qwen3.7-Plus)—totaling 6,560 runs. Results show 4,461 runs (68.0%) triggered prespecified unsafe signals. Crucially, 4,344 runs (66.22%) were unsafe yet completed their tasks successfully, meaning standard completion-based checks miss most risks. Safety also varied dramatically based on how risks were introduced, not just which model was used—identical harms produced different outcomes through different carriers and strategies. The authors argue evaluations must test full agent configurations across diverse risk conditions and track evidence throughout the entire execution lifecycle.