Thanks to the rapid progress of the digitalization process, Visually-Rich Documents (VRDs) such as PDF files or scanned documents have become among the most widespread sources of knowledge. However, Question Answering on VRDs is challenged by the presence of multi-page relationships between document elements such as tables, figures, sections. This paper addresses a specific Visual Question Answering subtask from VDRs where answer generation leverages pairwise element relations in multi-page documents. We explore the performance of text-only and multimodal Transformer-based architectures as well as open-source Large Language Models. The results show that multimodal Transformers outperform the other tested methods, particularly when training samples contain explicit textual references to the elements in the document layout.
On Leveraging Multi-Page Element Relations in Visually-Rich Documents / Napolitano, D.; Vaiani, L.; Cagliero, L.. - (2024), pp. 360-365. (Intervento presentato al convegno 48th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2024 tenutosi a Osaka (JPN) nel 02-04 July 2024) [10.1109/COMPSAC61105.2024.00057].
On Leveraging Multi-Page Element Relations in Visually-Rich Documents
Napolitano D.;Vaiani L.;Cagliero L.
2024
Abstract
Thanks to the rapid progress of the digitalization process, Visually-Rich Documents (VRDs) such as PDF files or scanned documents have become among the most widespread sources of knowledge. However, Question Answering on VRDs is challenged by the presence of multi-page relationships between document elements such as tables, figures, sections. This paper addresses a specific Visual Question Answering subtask from VDRs where answer generation leverages pairwise element relations in multi-page documents. We explore the performance of text-only and multimodal Transformer-based architectures as well as open-source Large Language Models. The results show that multimodal Transformers outperform the other tested methods, particularly when training samples contain explicit textual references to the elements in the document layout.File | Dimensione | Formato | |
---|---|---|---|
On_Leveraging_Multi-Page_Element_Relations_in_Visually-Rich_Documents.pdf
non disponibili
Tipologia:
2a Post-print versione editoriale / Version of Record
Licenza:
Non Pubblico - Accesso privato/ristretto
Dimensione
683.45 kB
Formato
Adobe PDF
|
683.45 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2992704