Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.
Entropic Score Metric: Decoupling Topology and Size in Training-Free NAS / Cavagnero, Niccolo'; Robbiano, Luca; Pistilli, Francesca; Caputo, Barbara; Averta, GIUSEPPE BRUNO. - ELETTRONICO. - (2023), pp. 1451-1460. (Intervento presentato al convegno IEEE/CVF International Conference on Computer Vision tenutosi a Paris (FR) nel 02-06 October 2023) [10.1109/ICCVW60793.2023.00158].
Entropic Score Metric: Decoupling Topology and Size in Training-Free NAS
Cavagnero Niccolo';Robbiano Luca;Pistilli Francesca;Caputo Barbara;Averta Giuseppe
2023
Abstract
Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.File | Dimensione | Formato | |
---|---|---|---|
Entropic Score metric Decoupling Topology and Size in Training-free NAS.pdf
accesso aperto
Tipologia:
2. Post-print / Author's Accepted Manuscript
Licenza:
PUBBLICO - Tutti i diritti riservati
Dimensione
362.77 kB
Formato
Adobe PDF
|
362.77 kB | Adobe PDF | Visualizza/Apri |
Entropic_Score_metric_Decoupling_Topology_and_Size_in_Training-free_NAS.pdf
non disponibili
Tipologia:
2a Post-print versione editoriale / Version of Record
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
937.39 kB
Formato
Adobe PDF
|
937.39 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2982841