Why Some AI Teachers Are Better Than Others, Explained

Researchers studied a widely used setup for studying learning: a "teacher" neural network generates training labels so a "student" network can learn the same function. Usually the teacher's structure is overlooked by assuming randomly-generated, normally-distributed parameters — which hides big variation in how learnable different teachers are. Formalizing learnability as the success rate of converging to the global minimum, the authors identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show these produce markedly different success rates across a large range of settings and activation functions. Digging into the loss landscape of small networks, they find two kinds of suboptimal local minima — out-of-bounds minima at the edge of the data distribution and interior minima — and, assuming infinite data and a fast readout layer, analytically reduce that landscape to two dimensions, showing the region of attraction of interior minima shifts with teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more out-of-bounds minima. Motivated by this, they show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases raises success rates — a step toward closing the gap between teacher-student studies and the more structured functions that arise in practice.