This paper presents a mixed reality system that, using the sensors mounted on the Microsoft Hololens headset and a cloud service, acquires and processes in real-time data to detect and track different kinds of objects and finally superimposes geographically coherent holographic texts on the detected objects. Such a goal has been achieved dealing with the intrinsic headset hardware limitations, by performing part of the overall computation in an edge/cloud environment. In particular, the heavier object detection algorithms, based on Deep Neural Networks (DNNs), are executed in the cloud. At the same time, we compensate for cloud transmission and computation latencies by running light scene detection and object tracking onboard the headset. The proposed pipeline allows meeting the real-time constraint by exploiting at the same time the power of state of art DNNs and the potential of Microsoft Hololens. This paper presents the design choices and describes the original algorithmic steps we devised to achieve real-time tracking in mixed reality. Finally, the proposed system is experimentally validated.

Real-time object detection and tracking in mixed reality using Microsoft HoloLens / Farasin, Alessandro; Peciarolo, Francesco; Grangetto, Marco; Gianaria, Elena; Garza, Paolo. - ELETTRONICO. - 4:(2020), pp. 165-172. ((Intervento presentato al convegno VISIGRAPP 2020 - 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications tenutosi a Valletta (MT) nel 27/02/2020 - 29/02/2020.

Real-time object detection and tracking in mixed reality using Microsoft HoloLens

Alessandro Farasin;Marco Grangetto;Paolo Garza
2020

Abstract

This paper presents a mixed reality system that, using the sensors mounted on the Microsoft Hololens headset and a cloud service, acquires and processes in real-time data to detect and track different kinds of objects and finally superimposes geographically coherent holographic texts on the detected objects. Such a goal has been achieved dealing with the intrinsic headset hardware limitations, by performing part of the overall computation in an edge/cloud environment. In particular, the heavier object detection algorithms, based on Deep Neural Networks (DNNs), are executed in the cloud. At the same time, we compensate for cloud transmission and computation latencies by running light scene detection and object tracking onboard the headset. The proposed pipeline allows meeting the real-time constraint by exploiting at the same time the power of state of art DNNs and the potential of Microsoft Hololens. This paper presents the design choices and describes the original algorithmic steps we devised to achieve real-time tracking in mixed reality. Finally, the proposed system is experimentally validated.
978-989-758-402-2
File in questo prodotto:
File Dimensione Formato  
VISAPP_2020_21_postprint.pdf

non disponibili

Descrizione: Articolo principale
Tipologia: 2a Post-print versione editoriale / Version of Record
Licenza: Non Pubblico - Accesso privato/ristretto
Dimensione 4.04 MB
Formato Adobe PDF
4.04 MB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11583/2823713