Deep reinforcement learning to explore multi-energy systems: a methodological approach

This work investigates the application of Deep Reinforcement Learning (DRL) to optimize energy flows in a multi-energy system (MES). The system integrates electricity, heat and gas grids and includes various energy storage and conversion technologies that enable energy exchange across sectors. While this integration increases flexibility and improves the use of non-dispatchable renewables, it also introduces significant complexity, making it difficult to define optimal control strategies a priori. Several optimization approaches have been proposed in the literature, ranging from metaheuristic methods to traditional deterministic models. DRL is emerging as a promising alternative due to its ability to handle nonlinear, high-dimensional problems without requiring structural simplifications. However, DRL generates black-box solutions where the internal logic of the learned policy is not directly interpretable. To understand the actual synergies within the MES and to extract meaningful control logic, it is necessary to analyze the policies learned by the DRL agent and uncover the underlying principles that govern its decisions. In this study, DRL is used as an exploratory tool to optimize the operation of an MES. The same scenario is first solved with a rule-based control algorithm based on efficiency prioritization. The DRL policy is then analyzed with interpretable surrogate models that allow us to extract the key rules and operational logic behind the agent’s decisions. This process transforms the DRL outcome from a black box solution into a deterministic control strategy where the interactions between technologies are explicitly known. This makes it possible to identify the most effective sectoral synergies and define an operational optimum for the given scenario. This combined approach thus offers both the performance advantages of DRL and the transparency of rule-based control.