AI Safety

Looking for papers on general formalizations of "agency"

Two papers propose new ways to detect agency without hand-labeling in dynamical systems

Deep Dive

A LessWrong user, lovagrus, is researching a general formal definition of agency that could support an operational detection across domains. They note that many definitions rely on specific terms like 'goal', 'intention', 'sensor', which are less useful for automatically finding agent-like structures in arbitrary dynamical systems. They are interested in the embedded agency framing and have found two valuable papers.

The first paper, 'Semantic information, autonomous agency, and nonequilibrium statistical physics' (2018), formalizes 'semantic information' — the information a system has about its environment that is causally necessary for viability. This could enable automatic detection by searching for system/environment decompositions with high concentration of such information. The second paper, 'Towards information based spatiotemporal patterns as a foundation for agent representation in dynamical systems' (2016), addresses the tracking problem: agents can change material parts but maintain organization. It proposes using integrated spatiotemporal patterns in dynamic Bayesian networks to define the same object over time. The user plans to follow citations and is especially interested in work tested in simulations like cellular automata or artificial life systems.

Key Points
  • The user seeks a formalism for agency that works across domains without pre-labeled agent definitions.
  • Two key papers: one (2018) on semantic information for viability detection, another (2016) on spatiotemporal patterns for tracking agents.
  • Goal: automatically detect agent-like structures in arbitrary dynamical systems like cellular automata or game worlds.

Why It Matters

A universal agency detector could enable AI to identify and track agents in complex systems, advancing embedded agency research.