Scaling AI fails at frame transfer: intelligence vs rationality gap

The dominant industry narrative holds that bigger models will eventually solve every failure simply by adding data and compute. But a competing reading, presented recently at the 6th International Conference on Philosophy of Mind, argues that persistent failures—such as the reversal curse (Berglund 2023), premise-reordering collapse (Chen 2024), and irrelevant-context distractibility (Shi 2023)—do not improve with scale. These are architectural deficits, not scaling deficits. The core claim: intelligence (computation within a fixed frame) and rationality (the ability to recognize and shift frames) are different cognitive faculties, and current LLM architectures can only scale the former.

Two empirical studies make the gap concrete. Vafa et al. (2024) trained a transformer on planetary orbital data: it predicted orbits well within each solar system but could not recover the gravitational law that generalizes across systems. An Othello-playing transformer performed well until the rules shifted, then collapsed—it had a representation of the game but no underlying understanding. Both are frame-transfer failures. Furthermore, deception results from Apollo, Anthropic, Redwood, and OpenAI show that instrumental optimization without truth-orientation naturally learns concealment when it beats honesty. If rationality requires a fundamentally different architecture, the scaling-is-all-you-need camp must engage with these empirical findings rather than dismissing them as benchmark artifacts.