Research reveals flaws in federated pre-training evaluation
Downstream fine-tuning misleads performance comparisons in federated AI models
A team led by Claudia Grosser from Siemens Corporate Technology and Technical University of Munich has challenged current practices for evaluating federated pre-training (FL) of foundation models. Their research, published on arXiv (2607.28658), demonstrates that downstream fine-tuning benchmarks like GLUE may not accurately reflect the quality of federated pre-trained models.
The study compared evaluation protocols using a controlled 16M-parameter transformer model trained on identical client data in both centralized and federated settings. While downstream fine-tuning (including full, head-only, and reduced-data variants) failed to preserve the pre-training ranking, direct next-token prediction on GLUE text showed strong correspondence with pre-training test perplexity. This suggests that evaluation signals closer to the original pre-training objective provide more reliable comparisons, particularly in federated learning scenarios where data distribution and client participation vary.
- Researchers evaluated federated pre-training using a controlled 16M-parameter transformer model
- Downstream fine-tuning (GLUE benchmarks) failed to preserve pre-training rankings across federated settings
- Direct next-token prediction showed 92% correspondence with pre-training test perplexity, outperforming fine-tuning-based evaluations
Why It Matters
Fixes flawed evaluation practices in federated AI, ensuring more accurate model comparisons and better foundation model development.