Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.

Entropic Score Metric: Decoupling Topology and Size in Training-Free NAS / Cavagnero, Niccolo'; Robbiano, Luca; Pistilli, Francesca; Caputo, Barbara; Averta, GIUSEPPE BRUNO. - ELETTRONICO. - (2023), pp. 1451-1460. (Intervento presentato al convegno IEEE/CVF International Conference on Computer Vision tenutosi a Paris (FR) nel 02-06 October 2023) [10.1109/ICCVW60793.2023.00158].

Entropic Score Metric: Decoupling Topology and Size in Training-Free NAS

Cavagnero Niccolo';Robbiano Luca;Pistilli Francesca;Caputo Barbara;Averta Giuseppe
2023

Abstract

Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typical of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and computational resources to generate accurate models. To overcome these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations; ii) a cyclic search algorithm to separately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour, resulting in the fastest and most accurate NAS method for ImageNet classification.
2023
979-8-3503-0744-3
File in questo prodotto:
File Dimensione Formato  
Entropic Score metric Decoupling Topology and Size in Training-free NAS.pdf

accesso aperto

Tipologia: 2. Post-print / Author's Accepted Manuscript
Licenza: PUBBLICO - Tutti i diritti riservati
Dimensione 362.77 kB
Formato Adobe PDF
362.77 kB Adobe PDF Visualizza/Apri
Entropic_Score_metric_Decoupling_Topology_and_Size_in_Training-free_NAS.pdf

non disponibili

Tipologia: 2a Post-print versione editoriale / Version of Record
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 937.39 kB
Formato Adobe PDF
937.39 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2982841