Why AI Keeps Misreading Your Instructions — New Research Explains

In a paper to appear in Topics in Cognitive Science, David M. Abel and Mark K. Ho draw attention to goals as representations and their content. In both cognitive science and computer science, they note, goals are conceptualized as cognitive states that flexibly combine with world knowledge to organize and specify purposeful behavior — making goals compositional representations whose content relates to rational behavior. That framing, the authors write, highlights a parallel with other areas of cognitive science, in particular the syntax-semantics interface in linguistics and logic, while foregrounding foundational questions about the expressivity, design, and efficiency of different goal representations. Goals are typically taken as fixed and as imposing constraints on desirable behavior, but the authors point to constraints on goal representations themselves — for example, whether a particular goal language is expressive enough to capture behaviors of interest, or whether different goal representations capture the same behavior. They synthesize work that aims to characterize the properties of different goal representations, suggesting these are points in a broader design space. They close by discussing how distinguishing the form and meaning of goals can elucidate the implicit assumptions we make about goals, inform the study of interactions between higher-level cognition and motivation, and isolate axes of variation for different conceptions of goals.