A server determines a plurality of faceprints representing a plurality of users to be recognized at a client device. Each faceprint contains a number of reference images for a given user that are used to recognize facial images of the user detecting in media captured at the client device. The faceprints delivered to the client device are determined for the client device based on the users likely to be detected in images captured at the client device. The reference images with a given faceprint delivered to the client device are selected by the server based on their recognition value in identifying the users likely to be detected in images captured at the client device.