We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits.

What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations / Plizzari, Chiara; Perrett, Toby; Caputo, Barbara; Damen, Dima. - (2023), pp. 13610-13620. (Intervento presentato al convegno International Conference on Computer Vision 2023 tenutosi a Paris (FR) nel 01-06 October 2023) [10.1109/ICCV51070.2023.01256].

What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations

Chiara Plizzari;Barbara Caputo;
2023

Abstract

We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits.
2023
979-8-3503-0718-4
File in questo prodotto:
File Dimensione Formato  
02272 (1).pdf

accesso aperto

Descrizione: What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
Tipologia: 2. Post-print / Author's Accepted Manuscript
Licenza: PUBBLICO - Tutti i diritti riservati
Dimensione 9.8 MB
Formato Adobe PDF
9.8 MB Adobe PDF Visualizza/Apri
What_can_a_cook_in_Italy_teach_a_mechanic_in_India_Action_Recognition_Generalisation_Over_Scenarios_and_Locations.pdf

non disponibili

Tipologia: 2a Post-print versione editoriale / Version of Record
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 10.47 MB
Formato Adobe PDF
10.47 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2981185