Images are loaded with semantic information that pertains to real-world ontologies: dog breeds share mammalian similarities, food pictures are often depicted in domestic environments, and so on. However, when training machine learning models for image classification, the relative similarities amongst object classes are commonly paired with one-hot-encoded labels. According to this logic, if an image is labelled as spoon, then tea-spoon and shark are equally wrong in terms of training loss. To overcome this limitation, we explore the integration of additional goals that reflect ontological and semantic knowledge, improving model interpretability and trustworthiness. We suggest a generic approach that allows to derive an additional loss term starting from any kind of semantic information about the classification label. First, we show how to apply our approach to ontologies and word embeddings, and discuss how the resulting information can drive a supervised learning process. Second, we use our semantically enriched loss to train image classifiers, and analyse the trade-offs between accuracy, mistake severity, and learned internal representations. Finally, we discuss how this approach can be further exploited in terms of explainability and adversarial robustness.
Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers / Perotti, Alan; Bertolotto, Simone; Pastor, Eliana; Panisson, André. - 1902:(2023), pp. 525-548. (Intervento presentato al convegno World Conference on eXplainable Artificial Intelligence. xAI 2023 tenutosi a Lisboa (Portugal) nel July 26–28, 2023) [10.1007/978-3-031-44067-0_27].
Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers
Bertolotto, Simone;Pastor, Eliana;
2023
Abstract
Images are loaded with semantic information that pertains to real-world ontologies: dog breeds share mammalian similarities, food pictures are often depicted in domestic environments, and so on. However, when training machine learning models for image classification, the relative similarities amongst object classes are commonly paired with one-hot-encoded labels. According to this logic, if an image is labelled as spoon, then tea-spoon and shark are equally wrong in terms of training loss. To overcome this limitation, we explore the integration of additional goals that reflect ontological and semantic knowledge, improving model interpretability and trustworthiness. We suggest a generic approach that allows to derive an additional loss term starting from any kind of semantic information about the classification label. First, we show how to apply our approach to ontologies and word embeddings, and discuss how the resulting information can drive a supervised learning process. Second, we use our semantically enriched loss to train image classifiers, and analyse the trade-offs between accuracy, mistake severity, and learned internal representations. Finally, we discuss how this approach can be further exploited in terms of explainability and adversarial robustness.File | Dimensione | Formato | |
---|---|---|---|
2308.00607.pdf
accesso riservato
Descrizione: Preprint
Tipologia:
1. Preprint / submitted version [pre- review]
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
3.44 MB
Formato
Adobe PDF
|
3.44 MB | Adobe PDF | Visualizza/Apri Richiedi una copia |
978-3-031-44067-0_27.pdf
accesso riservato
Descrizione: Post-print versione editoriale
Tipologia:
2a Post-print versione editoriale / Version of Record
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
4.72 MB
Formato
Adobe PDF
|
4.72 MB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2983327