Progressive² co-evolving distillation achieves major compression for large AI models

Knowledge distillation (KD) typically transfers knowledge from a large teacher model to a smaller student model, but performance degrades sharply when the capacity gap between server and client is huge. To address this, researchers from the paper "Progressive²" propose a novel teacher-student co-evolving framework. On the teacher side, instead of distilling all layers at once, Progressive² progressively adds layers in a raw-to-rich semantic order, creating a systematic learning curriculum. This staged approach helps the student absorb features gradually rather than being overwhelmed by differences in abstraction levels. The teacher also uses a multi-feature fusion adapter, theoretically grounded in Lipschitz continuity, to improve training stability and preserve rich representations.

On the student side, the method progressively reduces network size, allowing the student to co-evolve with the teacher iteratively. Rather than directly training a tiny model, this gradual shrinkage enables smoother adaptation and better final accuracy. Progressive² is designed as a flexible framework: the progressive teacher strategy can be deployed independently to balance accuracy and training efficiency, while the full teacher-student integration delivers further performance improvements. This approach offers a practical solution for substantial model compression, particularly in scenarios where a large gap exists between server-side models and client-device requirements, such as edge AI and real-time inference applications.