Toward a Realistic Benchmark for Out-of-Distribution Detection

Deep neural networks are increasingly used in a wide range of technologies and services, but remain highly susceptible to out-of-distribution (OOD) samples, that is, samples drawn from a different distribution from the original training set. A popular way to tackle this issue is to endow deep neural networks with the ability to detect OOD samples. Several benchmarks have been proposed to design and validate OOD techniques. However, many rely on far-OOD samples drawn from very different distributions, and thus lack the complexity needed to capture the nuances of real-world scenarios. In this work, we introduce a comprehensive benchmark for OOD detection, based on ImageNet and Places365, that assigns individidual class as in-distribution or out-of-distribution depending on the semantic similarity with the training set. Experimental results on different OOD detection techniques highlight how their measured effectiveness depends on the selected benchmark, and that confidence-based techniques may outperform classifier-based ones on near-OOD samples.