Early Reliability Assessment of AI-based Automotive Systems

The availability of powerful Artificial Intelligence (AI) algorithms boosts the development of advanced functionalities in the automotive domain and is essential to enable the deployment of autonomous and semi-autonomous decision-making vehicles. However, integrating such advanced and complex functionalities in automotive systems is challenging due to several factors, including: i) the mandatory compliance with strict safety regulations, which require effective strategies to ensure timely development while allowing thorough dependability evaluations, and ii) the short time-to-market imposing limited development, verification, and validation periods. In particular, the analysis of the effects of faults affecting the hardware executing an AI-based application is made challenging by the target system’s complexity (in terms of both hardware and software). Reliability analysis often resorts to Fault Injection techniques. However, Fault Injection experiments are often unacceptably time-consuming and limited to some components of the overall system, thus failing to consider the fault impact at the vehicle level. This work proposes a new method, named Two-steps IntegrAted Reliability Assessment (TIARA), for early estimation of the impact at the vehicle level of faults affecting the hardware running AI-based perception tasks in the automotive domain. TIARA allows for the early exploration and evaluation of algorithms, driving agents, and critical operational scenarios. TIARA can estimate the effects of faults affecting a subsystem up to the vehicle level, integrating a fault injection approach at the neural network level with a commercial automotive-grade virtual scenario generator. When compared to previous works, TIARA dramatically reduces the required computational effort by adopting a two-stage evaluation strategy. It first performs static analysis to determine fault vulnerabilities and identify the most vulnerable parts (code blocks) in a targeted application. Then, it focuses on the most susceptible parts of the neural network and estimates system-level effects on vehicle dynamics by combining the system’s perception, control, and driving features. We validated our methodology through the exhaustive evaluation of two applications: Lane Centering Assistance (LCA) and Emergency Lane Keeping Assistance (ELKA), using the YoloP model for perception. The experimental results on nine different driving scenarios show that TIARA allows for an effective early estimation of systems reliability through relevant driving dynamics and comfort metrics, as mandated by standards, while reducing computing complexity by up to 43.2X in comparison with a fully-exhaustive evaluation approach. In addition, the validation of the TIARA methodology through a hardware-in-the-loop implementation shows that the results closely match the behavior of a real-world system, demonstrating the versatility of the TIARA strategy for the evaluation of automotive systems.