Multimodal Fusion of Face and Gait for Person Identification in Automotive Applications

Smart and secure access to vehicles is a crucial aspect of the evolving automotive industry. This paper focuses on the development of an end-to-end multimodal biometric recognition framework that identifies people walking towards a vehicle from an RGB video feed. The framework is based on a deep-learning pipeline for person detection and tracking, face and gait feature extraction, and fusion of the two modalities at the score and feature level. Traditional face recognition systems can suffer from variations in lighting and occlusions. In order to deal with these issues, the proposed framework integrates face and gait features with the aim to enhance accuracy. The pipeline is modular, enabling seamless integration of new models for each step of person identification without the need for additional training. Baseline face and gait recognition models, as well as score- and feature-level fusion techniques are evaluated on subsets of the CASIA-A and CASIA-B datasets. Experimental results show that weighted mean score-level fusion significantly improves both Rank-1 accuracy and verification accuracy (TAR@FAR=10^-5) over unimodal baselines. Overall, reported work provides insights into current limitations and suggests directions for future research about secure identity verification in vehicles.