We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits.
What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations / Plizzari, Chiara; Perrett, Toby; Caputo, Barbara; Damen, Dima. - (2023), pp. 13610-13620. (Intervento presentato al convegno International Conference on Computer Vision 2023 tenutosi a Paris (FR) nel 01-06 October 2023) [10.1109/ICCV51070.2023.01256].
What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
Chiara Plizzari;Barbara Caputo;
2023
Abstract
We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits.File | Dimensione | Formato | |
---|---|---|---|
02272 (1).pdf
accesso aperto
Descrizione: What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
Tipologia:
2. Post-print / Author's Accepted Manuscript
Licenza:
PUBBLICO - Tutti i diritti riservati
Dimensione
9.8 MB
Formato
Adobe PDF
|
9.8 MB | Adobe PDF | Visualizza/Apri |
What_can_a_cook_in_Italy_teach_a_mechanic_in_India_Action_Recognition_Generalisation_Over_Scenarios_and_Locations.pdf
non disponibili
Tipologia:
2a Post-print versione editoriale / Version of Record
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
10.47 MB
Formato
Adobe PDF
|
10.47 MB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2981185