Human-Human Joint Interaction in Cross-Reality Systems: The Impact of Immersion and Synchronicity

Performing a Joint Interaction (JI) across the Reality–Virtuality (R-V) continuum remains challenging particularly in enabling shared manipulation of tangible real-world entities from distinct realities. To address this gap, this work investigates how interaction synchronicity and immersion level shape Cross-Reality (CR) JI involving shared manipulation of tangible entities by focusing on their impact on aspects pertaining task perceived presence, perceived workload, sense of inclusion/exclusion, empowerment and perceived control. To support this investigation, an exploratory user study examined human-human CR JI scenarios through the development of three interaction probes: a checkerboard, a comic book, and a music player, each re-creating specific types of JI that enables an Augmented Reality (AR) user and a real-world user to engage with a shared, manipulable entity under varying synchronization conditions. The quantitative results suggest that synchronous interaction enhanced perceived presence, while qualitative accounts pointed to a stronger inclusion and connection. Perceived workload showed differences across immersion conditions, with qualitative findings further highlighting the role of interaction constraints and control distribution in shaping participants’ experiences. Overall, effective CR JI design requires balancing physical and virtual affordances while enabling meaningful participation across asymmetric realities. These insights are synthesized into initial design implications for future CR JI systems, providing guidance for interaction approaches that support meaningful participation, awareness, and coordination across heterogeneous realities.