Semantic-aware decoding of covert inner speech: A multimodal EEG–EMG–audio framework

Non-invasive brain–computer interfaces (BCIs) aim to restore communication by decoding intended messages directly from neural activity, even without audible speech. However, evidence remains limited that non-invasive signals can support subject-held-out decoding of predefined semantic intentions from covert inner speech. We investigate whether a semantic-aware multimodal framework can generalize from overt spoken commands to covert inner speech using scalp electroencephalography (EEG), auxiliary electromyography (EMG), and overt-speech audio supervision. Ten healthy participants produced four everyday commands (water, toilet, light, pain) in overt and covert phases. Overt trials included synchronized EEG, EMG, and audio, whereas covert trials were evaluated without audio using EEG with auxiliary EMG. The proposed model learns a shared latent representation from overt multimodal supervision and covert training trials from non-held-out subjects, combining supervised contrastive learning, ArcFace classification, semantic prototype alignment, and overt–covert regularization. Evaluation follows a subject-held-out covert protocol with target-subject overt calibration: covert trials from the held-out participant are never used for training, model selection, or calibration. Across subjects, the model achieves a mean overt-validation accuracy of 0.54 and a mean covert-test accuracy of 0.42 on the four-class task, above the 0.25 chance level. Latent-space and semantic-retrieval analyses indicate that the learned representations retain class-related structure and alignment with text-based semantic prototypes. Ablations reveal that multimodal overt information supports within-subject decoding and representational analysis, whereas EEG-only overt supervision is more robust under held-out-subject covert transfer. These results identify subject variability, modality transfer, and calibration as key challenges for future semantic BCIs.