State-of-the-art speech models may exhibit suboptimal performance in specific population subgroups. Detecting these challenging cohorts is crucial for enhancing model robustness and fairness. Traditional methods for subgroup identification typically rely on demographic information such as age, gender, and origin. However, collecting such sensitive data at deployment time can be impractical or unfeasible due to privacy concerns.This paper introduces a novel Problematic Subgroup Identification model (PSI) to (i) automatically predict if an utterance belongs to problematic subgroups and (ii) provide an interpretable representation of these subgroups. PSI exploits confidence models (CMs) to encode information about sources of errors. CM fine-tuning based on problematic subgroup identification techniques allows accurate subgroup identification. PSI leverages demographic features only during its training, avoiding the need for sensitive data collection at deployment time. Experimental results on automatic speech recognition and intent classification datasets show PSI’s effectiveness in both identifying challenging subgroups and providing an interpretable subgroup description. These findings underscore the potential of PSI as a valuable tool for improving the robustness and fairness of speech models in real-world applications.
Leveraging confidence models for identifying challenging data subgroups in speech models / Koudounas, Alkis; Pastor, Eliana; Mazzia, Vittorio; Giollo, Manuel; Gueudre, Thomas; Reale, Elisa; Attanasio, Giuseppe; Cagliero, Luca; Cumani, Sandro; de Alfaro, Luca; Baralis, Elena; Amberti, Daniele. - ELETTRONICO. - (2024), pp. 134-138. (Intervento presentato al convegno 2024 IEEE International Conference on Acoustics, Speech and Signal Processing Workshop (ICASSPW) tenutosi a Seoul (KOR) nel 14-19 April, 2024) [10.1109/ICASSPW62465.2024.10626001].
Leveraging confidence models for identifying challenging data subgroups in speech models
Koudounas, Alkis;Pastor, Eliana;Mazzia, Vittorio;Gueudre, Thomas;Attanasio, Giuseppe;Cagliero, Luca;Cumani, Sandro;de Alfaro, Luca;Baralis, Elena;
2024
Abstract
State-of-the-art speech models may exhibit suboptimal performance in specific population subgroups. Detecting these challenging cohorts is crucial for enhancing model robustness and fairness. Traditional methods for subgroup identification typically rely on demographic information such as age, gender, and origin. However, collecting such sensitive data at deployment time can be impractical or unfeasible due to privacy concerns.This paper introduces a novel Problematic Subgroup Identification model (PSI) to (i) automatically predict if an utterance belongs to problematic subgroups and (ii) provide an interpretable representation of these subgroups. PSI exploits confidence models (CMs) to encode information about sources of errors. CM fine-tuning based on problematic subgroup identification techniques allows accurate subgroup identification. PSI leverages demographic features only during its training, avoiding the need for sensitive data collection at deployment time. Experimental results on automatic speech recognition and intent classification datasets show PSI’s effectiveness in both identifying challenging subgroups and providing an interpretable subgroup description. These findings underscore the potential of PSI as a valuable tool for improving the robustness and fairness of speech models in real-world applications.File | Dimensione | Formato | |
---|---|---|---|
leveraging-confidence-models-for-identifying-challenging-data-subgroups-in-speech-models.pdf
accesso aperto
Descrizione: Leveraging Confidence Models for Identifying Challenging Data Subgroups in Speech Models
Tipologia:
2. Post-print / Author's Accepted Manuscript
Licenza:
Pubblico - Tutti i diritti riservati
Dimensione
194.1 kB
Formato
Adobe PDF
|
194.1 kB | Adobe PDF | Visualizza/Apri |
Leveraging_Confidence_Models_for_Identifying_Challenging_Data_Subgroups_in_Speech_Models.pdf
accesso riservato
Tipologia:
2a Post-print versione editoriale / Version of Record
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
880.67 kB
Formato
Adobe PDF
|
880.67 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2986418