Australian AI Safety Forum warns of risks from agentic AI and loss of control

The Australian AI Safety Forum 2026, held at the University of Sydney on July 7-8, convened researchers, policymakers, industry practitioners, and civil society groups to address AI safety from multiple angles. The program was anchored by the 2026 International AI Safety Report, led by Turing Award winner Yoshua Bengio with contributions from over 100 experts and supported by over 30 countries. Key report themes shaped discussions: frontier models excel in math, coding, and science but fail at simple tasks; real-world harm from AI-driven scams, fraud, non-consensual imagery, and cyberattacks is escalating; reliability remains a major constraint as models fabricate information and produce flawed code, especially in longer agentic workflows; and loss of control remains an uncertain but potentially severe risk, with expert opinion divided. A layered risk management strategy was recommended, combining capability testing, technical safeguards, incident reporting, governance, and societal resilience.

Among standout sessions, an academic talk on evaluating agentic AI highlighted a critical gap: most benchmarks score single-prompt answers, but agentic systems that call tools, carry memory, and change external systems require trajectory-level evaluation. Behaviors like task completion, tool selection, permission use, recovery from failure, and downstream side effects are not captured by current benchmarks. This is especially relevant for enterprise AI, where a polished final answer can mask inefficiencies in the process. Another session on AI character drift under automated R&D explored recursive self-improvement risks. The forum underscored the inherent tension between waiting for certainty (which may leave us exposed) and acting early (which risks locking in ineffective controls).