MedOpenSeg: Open-World Medical Segmentation with Memory-Augmented Transformers

Open-world segmentation in medical imaging presents unique challenges, as models must generalize to seen and unseen classes while retaining knowledge of previously seen structures. We propose MedOpenSeg, a Memory-Augmented transformer framework that dynamically stores and updates class prototypes to enhance segmentation accuracy, improve adaptability to new anatomical structures, and detect novel regions during inference. MedOpenSeg integrates a Swin-Transformer 3D backbone with a memory bank module that retrieves class-specific feature embeddings and facilitates prototypebased novelty detection using cosine similarity and Euclidean Distance Sum (EDS). We benchmark MedOpenSeg on multiple datasets against state-of-the-art closed-set segmentation and foundation models, demonstrating its effectiveness in handling openset medical segmentation. Code is publicly available at https://github.com/robustmleurecom/MedOpenSeg.git.