Disclosed is a server for performing authentication or identification using biometric information including basic information and detailed information includes a storage for storing basic information and detailed information that are separately encrypted for each of a plurality of users, a communicator for communicating with an external device, and a processor configured to, based on separately encrypted basic information and detailed information being received from an external terminal device through the communicator, performing user authentication or user identification for the received basic information and detailed information by decrypting and comparing the stored encrypted basic information and the received encrypted basic information, and comparing the received detailed information with at least one piece of stored detailed information corresponding to a piece of basic information having a degree of similarity that is higher than or equal to a predetermined value and with the received basic information among the stored basic information.