Can GPT-2 models independently discover the concept of zero?

Researchers from Princeton and NYU (Zeng, Griffiths, Lake) tested whether small language models (GPT-2 scale) can independently discover the mathematical concept of zero using simple arithmetic tasks. The models were evaluated on their ability to generalize to zero-valued operands without ever seeing zero during training—a strong test of out-of-distribution reasoning. Results showed that zero-shot, these models completely failed to produce meaningful outputs for zero-involving equations. However, after fine-tuning on just 10–100 examples involving zero, performance improved substantially. Interestingly, models pretrained on natural language required roughly 50% fewer examples than non-pretrained counterparts, suggesting that language abilities can scaffold mathematical discovery in neural networks.

The study underscores a fundamental limitation: current transformer-based models cannot spontaneously invent new mathematical structures (like zero) purely from arithmetic training. This aligns with the broader challenge of achieving true mathematical creativity in AI. The authors note that while language pretraining helps, it doesn't enable the kind of 'aha' moment that humans have when grasping zero's unique properties. For AI to push beyond human mathematical knowledge, architectures may need explicit mechanisms for concept invention, not just pattern extrapolation from data.