Eggs, rooms, puzzles, and talking about AI

AI researcher and philosopher Katja Grace published a viral essay on LessWrong detailing a real-world cognitive experiment: hiding 156 Easter eggs throughout a shared house to observe how people search. She discovered that searchers, despite conscious effort, consistently missed eggs placed in 'plain sight' on objects like a worn cushion or behind a familiar bathroom sign. This failure occurred because human perception inherently abstracts away details, simplifying a cushion into a 'square' or a sign into 'label,' making specific irregularities invisible. Grace argues this isn't a failure of attention but a fundamental feature of how intelligent systems—human or artificial—parse overwhelming sensory data.

The essay extrapolates this to a critical problem in AI development: LLMs like GPT-4o and Claude 3.5 Sonnet operate on vast but abstracted world models. They make decisions based on simplified representations of reality, which may omit crucial, exploitable details. Just as an egg-hider can predict and exploit human abstraction habits, a malicious actor could design prompts or scenarios that exploit an AI's blind spots, leading to unexpected failures. Grace connects this to the challenge of room allocation among housemates, showing that even simple problems become complex when our abstractions about 'rooms' and 'needs' break down. The piece suggests that for AI to be robust and safe, developers must build systems that can dynamically adjust their 'level of detail' or recognize when their abstractions are being manipulated.