Leveraging confidence models for identifying challenging data subgroups in speech models

State-of-the-art speech models may exhibit suboptimal performance in specific population subgroups. Detecting these challenging cohorts is crucial for enhancing model robustness and fairness. Traditional methods for subgroup identification typically rely on demographic information such as age, gender, and origin. However, collecting such sensitive data at deployment time can be impractical or unfeasible due to privacy concerns.This paper introduces a novel Problematic Subgroup Identification model (PSI) to (i) automatically predict if an utterance belongs to problematic subgroups and (ii) provide an interpretable representation of these subgroups. PSI exploits confidence models (CMs) to encode information about sources of errors. CM fine-tuning based on problematic subgroup identification techniques allows accurate subgroup identification. PSI leverages demographic features only during its training, avoiding the need for sensitive data collection at deployment time. Experimental results on automatic speech recognition and intent classification datasets show PSI’s effectiveness in both identifying challenging subgroups and providing an interpretable subgroup description. These findings underscore the potential of PSI as a valuable tool for improving the robustness and fairness of speech models in real-world applications.