A facility access control system and corresponding method are provided. The facility access control system includes a camera configured to capture an input image of a subject attempting to enter or exit a restricted facility. The facility access control system further includes a memory storing a deep learning model configured to perform multi-task learning for a pair of tasks including a liveness detection task and a face recognition task. The facility access control system also includes a processor configured to apply the deep learning model to the input image to recognize an identity of the subject in the input image regarding being authorized for access to the facility and a liveness of the subject. The liveness detection task is configured to evaluate a plurality of different distracter modalities corresponding to different physical spoofing materials to prevent face spoofing for the face recognition task.