Recognizing human emotions from videos requires a deep understanding of the underlying multimodal sources, including images, audio, and text. Since the input data sources are highly variable across different modality combinations, leveraging multiple modalities often requires ad hoc fusion networks. To predict the emotional arousal of a person reacting to a given video clip we present ViPER, a multimodal architecture leveraging a modality-agnostic transformer-based model to combine video frames, audio recordings, and textual annotations. Specifically, it relies on a modality-agnostic late fusion network which makes ViPER easily adaptable to different modalities. The experiments carried out on the Hume-Reaction datasets of the MuSe-Reaction challenge confirm the effectiveness of the proposed approach.

ViPER: Video-based Perceiver for Emotion Recognition / Vaiani, Lorenzo; LA QUATRA, Moreno; Cagliero, Luca; Garza, Paolo. - ELETTRONICO. - (In corso di stampa). ((Intervento presentato al convegno Multimodal Sentiment Analysis Challenge (MuSe 2022) tenutosi a Lisbon (PT) nel October 10-15, 2022 [10.1145/3551876.3554806].

ViPER: Video-based Perceiver for Emotion Recognition

Lorenzo Vaiani;Moreno La Quatra;Luca Cagliero;Paolo Garza
In corso di stampa

Abstract

Recognizing human emotions from videos requires a deep understanding of the underlying multimodal sources, including images, audio, and text. Since the input data sources are highly variable across different modality combinations, leveraging multiple modalities often requires ad hoc fusion networks. To predict the emotional arousal of a person reacting to a given video clip we present ViPER, a multimodal architecture leveraging a modality-agnostic transformer-based model to combine video frames, audio recordings, and textual annotations. Specifically, it relies on a modality-agnostic late fusion network which makes ViPER easily adaptable to different modalities. The experiments carried out on the Hume-Reaction datasets of the MuSe-Reaction challenge confirm the effectiveness of the proposed approach.
File in questo prodotto:
File Dimensione Formato  
MuSe2022_reaction_challenge.pdf

non disponibili

Descrizione: post-print authors version
Tipologia: 2. Post-print / Author's Accepted Manuscript
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 815.2 kB
Formato Adobe PDF
815.2 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2971099