Dynamic optimization of provider-based scheduling for HPC workloads

The vast array of cloud providers present in today’s market proffer a suite of High-Performance Computing (HPC) services. However, these offerings are characterized by significant variations in execution times and cost structures. Consequently, selecting the optimal cloud provider and configuring the features of the chosen computing instance (e.g. virtual machines) proves to be a challenging task for users intending to execute HPC workloads. This paper introduces a novel component designed for effortless integration with existing HPC scheduling systems. This module’s primary function is to facilitate the selection of the most appropriate cloud provider for each distinct job, thereby empowering dynamic and adaptive cost-minimization strategies. Through the application of data augmentation techniques and the employment of Continuous Machine Learning, the system is endowed with the capability to operate efficiently with cloud providers that have not been previously utilized. Furthermore, it is capable of tracking the evolution of jobs over time. Our results show that this component can achieve consistent economic savings, based on the quality of the data used in the training phase.