CPDP method cuts distributed training time 13.8% while boosting accuracy 2.44 points

A new paper from Imane Ettifouri, Mostapha Zbakh, and Claude Tadonki tackles a core bottleneck in distributed deep learning: communication overhead. Their Controlled Periodic Data Parallelism (CPDP) method modifies the standard DDP approach by reducing synchronization frequency while adding a reconciliation step that combines gradient AllReduce with SlowMo parameter averaging. This keeps worker trajectories aligned without the per-iteration sync cost. CPDP is implemented as a plug-in replacement for PyTorch DDP, making it easy to adopt.

Experiments on Grid'5000 across intra-site clusters and a cross-site WAN link (Nancy to Sophia, 16.6 ms RTT) show clear gains. At K=4 (synchronize every 4 steps), CPDP achieved 2.44 percentage points higher peak test accuracy on ResNet-50/CIFAR-100 with a 13.8% reduction in average wall-clock time compared to DDP. Direct profiling reveals exposed synchronization time at K=4 is roughly half that of DDP, explaining the improved accuracy-time trade-off. Additional tests on ViT-S/CIFAR-100 and ResNet-50/TinyImageNet confirm CPDP remains competitive with DDP and generally outperforms LocalSGD. The results establish synchronization frequency as a practical control parameter for communication-constrained distributed training, especially across WANs.