Understanding Surgical Triplet Videos Through Transferable Visual Models from Natural Language Supervision

Surgical triplet video understanding is essential for accu-rately recognizing and classifying surgical actions in real-time video data, enabling improved surgical planning and support. However, the recog-nition of surgical video triplets is particularly influenced by the varying frequency and duration of specific actions. Furthermore, the similarity in motion trajectories across various surgical actions presents another issue in video-based fine-grained surgical action recognition, complicat-ing the differentiation of similar actions. To address these challenges, we propose a new comprehensive paired image-text surgical activity event dataset (SAE), consisting of 90,500 pairs of images and text depict-ing surgical actions. Additionally, we introduce TriClip, a novel dual-branch contrastive multimo dalframework,whicheffectivelybridgesthegapbetweenvisualandtextualmodalitiesinsurgicalactionrecogni-tion.Byleveragingtransferablevisualmodelsfromnaturallanguagesupervision,TriClipwasevaluatedusingtheCholecT45dataset,whereitachievedanSOTAaverageprecisionof42.1%,settinganewstate-of-the-artinthefieldofsurgicalactionrecognition.