Refined Two-Sided Learning Rate Tuning for Robust Evaluation in Federated Learning

This paper investigates the impact of client and server learning rates on training deep neural networks in Federated Learning (FL). While previous research has primarily focused on optimizing the initial values of these learning rates, we demonstrate that this approach alone is insufficient for maximizing model performance and training efficiency. To address this weakness, we propose a revised two-sided learning rate optimization strategy that integrates learning rate decay schedules as tunable variables and adjusts the learning rate configurations based on the target training budget, allowing for more effective optimization. We conduct an extensive experimental evaluation to quantify the improvements offered by our approach. The results reveal that (i) integrating decay schedules into the tuning process leads to significant performance enhancements, and (ii) the optimal configuration of client-server decay schedules is strongly influenced by the training round budget. Based on these findings, we claim that performance evaluations of new FL algorithms should extend beyond the fine-tuning of the initial learning rate values, as done in the state-of-the-art approach, and include the optimization of decay schedules according to the available training budget.