Exploiting spatial and temporal consistency for fault detection in semantic segmentation NN for indoor and outdoor exploration

In safety-critical applications such as autonomous driving and robotics, the reliability of Deep Neural Networks (DNNs) performing video semantic segmentation is paramount. However, detecting hardware-induced faults in these complex tasks remains a challenge. State-of-the-art methods often evaluate fault criticality using overly conservative metrics like mean Intersection over Union (IoU) or Pixel Accuracy (PA). Furthermore, for online detection, they typically require invasive white-box access to internal states and miss transient faults by analyzing only single frames. To address these limitations, this work proposes a novel, unsupervised Spatio-Temporal Fault Detection framework. The proposed Single Frame and Temporal Consistency (SFTC) methodology leverages a dual-phase strategy: Single Frame Consistency (SFC), which evaluates geometric features (Area, Position, Symmetry, Shape) to identify structural degradations within single frames, and a Temporal Consistency (TC), which monitors frame-to-frame dynamics to detect transient faults that violate structural and visual continuity. To validate this approach, we present an extension of the Faulty Output Dataset (FOD), now covering both outdoor automotive scenarios (Fast-SCNN on Cityscapes) and indoor robotic environments (ESANet on NYU Depth V2). Experimental results for both permanent and transient faults demonstrate that the synergy between spatial and temporal monitoring significantly enhances reliability, achieving a detection rate increase of up to 9.14% compared to the state-of-the-art performance, by operating in a fully black-box way.