Extending upper limb user interactions in AR, VR and MR headsets employing a custom-made wearable device

Upper limb interactions play an important role in scenarios employing virtual, augmented, and mixed reality. Nu- merous sensors (e.g., optical, magnetic, mechanical, myography), have been employed to provide natural interactions, in some cases requiring additional hardware. Recently, virtual, augmented, and mixed reality headsets have started embedding sensors to avoid the use of external hardware that can be cumbersome, thus increasing their ease of use and setup. Microsoft’s recent approach to embedded sensors is known as inside/out tracking, comprised of integrated cameras that allow interactions through fingers and eye tracking with the Hololens, or controller tracking with the mixed reality headsets. However, the motion capture area of the inside/out sensors is limited to the field of view of the cameras. In this paper, we introduce a custom upper limb motion tracking device that extends the user interactions while employing a VR, AR, or MR headset. Our 3D motion tracking system is a compact wireless wearable prototype employing inertial measurement units that provide orientation and position data that provides upper limb user interaction outside the field of view of the inside/out sensors.