Mixed reality-based user interaction feedback for a hand-controlled interface targeted to robot teleoperation

The continuous progress in the field of robotics and the diffusion of its related application scenarios in today’s modern world makes human interaction and communication with robots an aspect of fundamental importance. The development of interfaces based on natural interaction paradigms is getting an increasingly captivating topic in Human-Robot Interaction (HRI), due to their intrinsic capabilities in providing ever more intuitive and effective control modalities. Teleoperation systems require to handle a non-negligible amount of information coming from on-board sensors as well as input devices, thus increasing the workload of remote users. This paper presents the design of a 3D User Interface (3DUI) for the control of teleoperated robotic platforms aimed at increasing the interaction efficiency. A hand gesture driven controller is used as input modality to naturally map the position and gestures of the user’s hand to suitable commands for controlling the platform components. The designed interface leverages on mixed reality to provide a visual feedback to the control commands issued by the user. The visualization of the 3DUI is superimposed to the video stream provided by an on-board camera. A user study confirmed that the proposed solution is able to improve the interaction efficiency by significantly reducing the completion time for tasks assigned in a remote reach-and-pick scenario.