Thanks to the rapid progress of the digitalization process, Visually-Rich Documents (VRDs) such as PDF files or scanned documents have become among the most widespread sources of knowledge. However, Question Answering on VRDs is challenged by the presence of multi-page relationships between document elements such as tables, figures, sections. This paper addresses a specific Visual Question Answering subtask from VDRs where answer generation leverages pairwise element relations in multi-page documents. We explore the performance of text-only and multimodal Transformer-based architectures as well as open-source Large Language Models. The results show that multimodal Transformers outperform the other tested methods, particularly when training samples contain explicit textual references to the elements in the document layout.

On Leveraging Multi-Page Element Relations in Visually-Rich Documents / Napolitano, D.; Vaiani, L.; Cagliero, L.. - (2024), pp. 360-365. (Intervento presentato al convegno 48th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2024 tenutosi a Osaka (JPN) nel 02-04 July 2024) [10.1109/COMPSAC61105.2024.00057].

On Leveraging Multi-Page Element Relations in Visually-Rich Documents

Napolitano D.;Vaiani L.;Cagliero L.
2024

Abstract

Thanks to the rapid progress of the digitalization process, Visually-Rich Documents (VRDs) such as PDF files or scanned documents have become among the most widespread sources of knowledge. However, Question Answering on VRDs is challenged by the presence of multi-page relationships between document elements such as tables, figures, sections. This paper addresses a specific Visual Question Answering subtask from VDRs where answer generation leverages pairwise element relations in multi-page documents. We explore the performance of text-only and multimodal Transformer-based architectures as well as open-source Large Language Models. The results show that multimodal Transformers outperform the other tested methods, particularly when training samples contain explicit textual references to the elements in the document layout.
2024
979-8-3503-7696-8
File in questo prodotto:
File Dimensione Formato  
On_Leveraging_Multi-Page_Element_Relations_in_Visually-Rich_Documents.pdf

non disponibili

Tipologia: 2a Post-print versione editoriale / Version of Record
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 683.45 kB
Formato Adobe PDF
683.45 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2992704