Toward Multimodal Audio Interfaces in Educational Music Production: A Microcontroller-Based Embedded System with Voice and Facial Control

Multimodal interaction represents a promising direction to enhance accessibility and inclusivity in Educational Music Production (EMP) environments. This work reports on the ongoing design, development, and feasibility evaluation of a microcontroller-based embedded system designed to extend standard audio interfaces with multimodal interaction capabilities. The proposed solution is architecturally composed of two main elements: an external stereo Analog-to-Digital Converter (ADC), specifically the PCM1803AEVM, which digitizes analog audio signals originating from standard audio interfaces; and a NUCLEO-H723ZG evaluation board, based on an STM32 microcontroller, which receives the digital audio stream, manages buffering, and enables data transmission over Ethernet. Once connected to a Local Area Network (LAN), the system operates as a self-contained server, hosting an onboard, browser-accessible Web Audio Interface (WAI) which allows the user to perform core production tasks through voice or facial commands, besides the standard manual ones. A functional evaluation with six users was conducted to assess the accuracy of the multimodal commands across diverse vocal and facial profiles. Results indicate promising recognition accuracy, supporting the case for further validation in representative EMP scenarios.