Risk-based Comprehensive Usability Evaluation of Software as a Medical Device

Introduction: Usability evaluation is a core aspect in risk assessment of medical devices, as it aims to ensure the device interface safety, avoiding that usability problems at interface level are not related to harm.Methods: Our research group applied our risk-based approach, international reference standards and guidelines to the usability evaluation of a large family of SaMD. The methodology used for the evaluation is an elaboration of regulatory prescriptions and is composed of a combination of quantitative and qualitative methods. In particular, the usability evaluation is structured in a two-stage evaluation composed by formative and summative evaluation. The formative stage is propaedeutic for the planning of the summative evaluation. The final assessment included the analysis of quantitative data collected through three questionnaires and a user test.Results and discussion: Risk-based task analysis led to the identification of the most common use error emerged during the user test performance. The three questionnaires led to different results: Heuristic analysis allowed the identification of violations to the heuristic principles as perceived by the users and their severity; SUS questionnaire provided an indicator of general device usability; the interview identified the usability problems of each device with respect to their functionalities.Conclusions: The study allowed the extensive assessment of the devices, the identification of usability issues, and the classification in terms of criticality of each issue. In conclusion the study led to different proposals to solve the issues and design changes.