AI's Next Frontier: Solving Ethics to Build the Best Future

A provocative essay argues that humanity cannot build the best possible future without first solving ethics—a problem so broad that it requires structured analysis. The author proposes that acting under moral uncertainty is manageable: when torn between maximizing and satisficing moral theories, we can allocate a tiny fraction of the universe to satisficing views (e.g., preserving homo sapiens) while dedicating the rest to maximizing theories (e.g., filling space with welfareans). This practical compromise sidesteps the maximizing-vs-satisficing debate, though the author notes plausible satisficing theories still favor maximizing The Good even if not morally obligatory.

More radically, the essay claims most moral disagreements could be eliminated by discovering facts. The fact-value distinction means we cannot derive 100% of 'ought' from 'is,' but nearly all apparent terminal value clashes are downstream of unresolved empirical questions. These include how to weight different experiences (tied to the hard problem of consciousness), whether personal identity is metaphysically meaningful, and which moral theories are even coherent given the nature of persons. The author suspects definitive answers exist in principle, and that AI—by modeling these questions or uncovering hidden facts—could help resolve moral uncertainty before deploying superintelligence, avoiding catastrophic misallocation of resources.