Theory Uplift Could Give Safety an Edge, But It's Being Ignored
Math AI is surging ahead of R&D, opening a safety window we're missing.
In a LessWrong post titled "theory uplift differentially benefits safety & is underleveraged," AI researcher Yudhister Kumar forecasts that near-superhuman mathematics AI will emerge by Q1 2027 (likely from OpenAI). This capability, he argues, is developing significantly faster than automated AI R&D, creating a temporary window where our ability to rigorously verify model behavior outpaces the creation of dangerous new capabilities. However, that window is being squandered.
Kumar points out that investment in theory-driven safety infrastructure is minuscule — roughly $100M across organizations like Convergent, Theorem, and ARIA, compared to an expected $37-100B/year in philanthropic funding for AI. He lists several theoretical frameworks that have advanced safety without boosting capabilities: singular learning theory (LLC estimators), computational mechanics (belief state geometry), and superposition/linear representation hypotheses (SAEs). Yet no one is building infrastructure to turn >$100M of compute credits into safety-relevant mathematical output. The bottleneck is institutional capacity, not capital.
- Near-superhuman math AI expected by Q1 2027, outpacing automated R&D by ~1.5 years.
- Only ~$100M invested in theory-driven safety vs. projected $37-100B/yr in AI philanthropy.
- Frameworks like singular learning theory and superposition hypothesis benefit safety without aiding capabilities.
Why It Matters
If safety theory isn't scaled now, the AI race could outpace our ability to control it.