Scheduling Latency-Sensitive Tasks in the Cloud Continuum with Hierarchical Reinforcement Learning

Service orchestrators such as Kubernetes are widely employed to automate the handling and scheduling of workloads, which involves determining the most suitable physical node on which to start a new task. The expanding application of Machine Learning (ML) algorithms, and in particular Reinforcement Learning (RL), opens up new development opportunities to make runtime decisions that can account for multiple metrics and varying network conditions. However, current RL-based solutions are unable to fit the growing complexity of distributed applications and infrastructure, characterized by a more heterogeneous resource continuum and the increasing need to minimize energy consumption while satisfying tasks' requirements. To fill this gap, we propose RL-ICE as an innovative scheduler that can work in such a cloud continuum by leveraging a multi-cluster and hierarchical RL to satisfy both user Quality of Experience (QoE) metrics and tenant's costs. We test RL-ICE in a simulated large-scale environment and in a real-world Kubernetes setup. In both scenarios, our solution effectively balances user-perceived latency, energy consumption, and deployment costs. Additionally, RL-ICE can dynamically respond to network failures by migrating microservices to maintain efficient management of resources.