An automatic computer vision pipeline for the in-line monitoring of freeze-drying processes

Infrared imaging sensors were proposed in the past years for the real-time monitoring and control of the Vacuum Freeze Drying (VFD) process. In order to extract reliable, real-time, and quantitative features from images collected by these sensors, a robust and automated image processing pipeline is required. Two major issues that have to be addressed concern the object detection and segmentation step, which locates and segments the objects whose temperature needs to be measured, and the tracking step, which is about following the movement of objects of interest to correlate information across subsequent images. Traditional intensity-based image analysis techniques are not reliable in this specific application, since the temperature, which is the intensity of a thermal picture, varies with time, whereas deep learning-based techniques are more robust to such changes. In this work, an object detector network based on a Faster Region Convolutional Neural Network (Faster R-CNN), and a Kernelized Correlation Filter (KCF) tracker were combined to monitor the VFD process of products contained in glass vials, which is a common scenario in the pharmaceutical domain. The object detector was trained on nine experimental acquisitions (7,981 images) and tested on nine more (7,201 images). Data augmentation methods consisting in the generation of synthetic sequences were used to effectively increase the performance while reducing the cost of data acquisition and annotation. The proposed technique achieved a recall and precision equal to 99.6%, with execution time compatible with a real-time application. The localization of the vials was precise enough to measure the mean temperature with an acceptable error.