Softmax-Driven Active Shape Model for Segmenting Crowded Objects in Digital Pathology Images

Automated segmentation of histological structures in microscopy images is a crucial step in computer-aided diagnosis framework. However, this task remains a challenging problem due to issues like overlapping and touching objects, shape variation, and background complexity. In this work, we present a novel and effective approach for instance segmentation through the synergistic combination of two deep learning networks (detection and segmentation models) with active shape models. Our method, called softmax-driven active shape model (SD-ASM), uses information from deep neural networks to initialize and evolve a dynamic deformable model. The detection module enables treatment of individual objects separately, while the segmentation map precisely outlines boundaries. We conducted extensive tests using various state-of-the-art architectures on two standard datasets for segmenting crowded objects like cell nuclei - MoNuSeg and CoNIC. The proposed SD-ASM consistently outperformed reference methods, achieving up to 8.93% higher Aggregated Jaccard Index (AJI) and 9.84% increase in Panoptic Quality (PQ) score compared to segmentation networks alone. To emphasize versatility, we also applied SD-ASMs to segment hepatic steatosis and renal tubules, where individual structure identification is critical. Once again, integration of SD-ASM with deep models enhanced segmentation accuracy beyond prior works by up to 6.2% in AJI and 38% decrease in Hausdorff Distance. The proposed approach demonstrates effectiveness in accurately segmenting touching objects across multiple clinical scenarios.