Crying jaywalker! Notifying take-over-requests and critical events in operational driving domain of autonomous vehicles via multimodal interfaces

The advent of self-driving cars promises to enable occupants to repurpose commuting time. However, although in conditional automation (SAE Level 3) drivers can engage in non-driving related tasks (NDRTs), they must be ready to intervene when prompted by the system with a take-over request (TOR). The vehicle may also need to warn the driver about critical events without notifying a TOR (as in sudden hard braking due to a jaywalker). Clearly and effectively communicating these events and their urgency is crucial for the successful adoption of autonomous vehicles. This work analyzes the impact of multimodal visual and audio cues in conveying this information. It considers an augmented reality (AR) windshield display (WSD) combining screen-fixed elements and world-registered AR overlays, alongside an auditory interface providing explanations and alerts through speech and abstract sounds. The effectiveness of these combined stimuli was evaluated through a user study conducted in a VR-based driving simulator.