Entropic Score Metric: Decoupling Topology and Size in Training-Free NAS

Neural Networks design is a complex and often daunting task, particularly for resource-constrained scenarios typi- cal of mobile-sized models. Neural Architecture Search is a promising approach to automate this process, but existing competitive methods require large training time and com- putational resources to generate accurate models. To over- come these limits, this paper contributes with: i) a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise en- tropy of its activations; ii) a cyclic search algorithm to sep- arately yet synergistically search model size and topology. Entropic Score shows remarkable ability in searching for the topology of the network, and a proper combination with LogSynflow, to search for model size, yields superior capa- bility to completely design high-performance Hybrid Trans- formers for edge applications in less than 1 GPU hour, re- sulting in the fastest and most accurate NAS method for Im- ageNet classification.