Federated Learning offers a powerful solution for training models on data that cannot be centrally stored due to privacy concerns. However, the existing paradigm suffers from high statistical heterogeneity across clients' data, resulting in client drift due to biased local solutions. This issue is particularly pronounced in the final classifier layer, severely impeding convergence speed during aggregation. To overcome these challenges, we introduce Federated Recursive Ridge Regression (Fed3R). This approach replaces the gradient-based classifier with a ridge regression-based classifier, computed in a closed form, ensuring client drift resilience and severely reducing convergence time and communication costs. The incremental formulation of Fed3R is equivalent to the ideal centralized ridge regression solution, enabling the utilization of more complex architectures with pre-trained parameters and robust generalization capabilities incompatible with previous federated learning techniques. We propose Fed3R in three variants, with Fed3R-RF significantly enhancing performance to levels akin to centralized training while remaining competitive regarding the total communication costs.

Fed3R: Recursive Ridge Regression for Federated Learning with strong pre-trained models / Fani', Eros; Camoriano, Raffaello; Caputo, Barbara; Ciccone, Marco. - ELETTRONICO. - 1:(2023), pp. 1-19. (Intervento presentato al convegno Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 tenutosi a New Orleans, LA (USA) nel Dec 16 2023).

Fed3R: Recursive Ridge Regression for Federated Learning with strong pre-trained models

Fani', Eros;Camoriano, Raffaello;Caputo, Barbara;Ciccone, Marco
2023

Abstract

Federated Learning offers a powerful solution for training models on data that cannot be centrally stored due to privacy concerns. However, the existing paradigm suffers from high statistical heterogeneity across clients' data, resulting in client drift due to biased local solutions. This issue is particularly pronounced in the final classifier layer, severely impeding convergence speed during aggregation. To overcome these challenges, we introduce Federated Recursive Ridge Regression (Fed3R). This approach replaces the gradient-based classifier with a ridge regression-based classifier, computed in a closed form, ensuring client drift resilience and severely reducing convergence time and communication costs. The incremental formulation of Fed3R is equivalent to the ideal centralized ridge regression solution, enabling the utilization of more complex architectures with pre-trained parameters and robust generalization capabilities incompatible with previous federated learning techniques. We propose Fed3R in three variants, with Fed3R-RF significantly enhancing performance to levels akin to centralized training while remaining competitive regarding the total communication costs.
File in questo prodotto:
File Dimensione Formato  
55_fed3r_recursive_ridge_regressi.pdf

accesso aperto

Tipologia: 2. Post-print / Author's Accepted Manuscript
Licenza: Pubblico - Tutti i diritti riservati
Dimensione 1.66 MB
Formato Adobe PDF
1.66 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2990259