On Detecting Anomalous TLS Connections with Artificial Intelligence Models

In recent years, anomaly-based intrusion detection systems using machine learning (ML) and deep learning techniques have started to be developed to mitigate cybersecurity attacks. An anomaly-based intrusion detection system performs traffic analysis by exploiting supervised or unsupervised ML algorithms and raises alerts if a suspicious pattern is encountered. In this paper, we exploit the Autoencoder neural network model to detect variants of a very famous attack discovered in 2014, namely Heartbleed. The attack was caused by an implementation flaw in the OpenSSL library, widely used in web servers, database systems, or e-mail servers to support the Transport Layer Security (TLS) protocol. To evaluate our model, we exploited the CIC-IDS2017 dataset and a custom one created on purpose. The proposed model recognized the anomalous TLS connections containing variants of the Heartbleed attack and distinguished them from the benign traffic in 85% of the cases.