A cropped bounding box selection operation is performed on a video captured by a video capture and playback system, to select one or more cropped bounding boxes from the video for processing by a face detection operation. The cropped bounding box selection operation identifies objects from the video images and assigns a ranking to each identified object based on certain priority criteria; one or more cropped bounding boxes corresponding to the objects with the highest ranking(s) are then processed by the face detection operation to detect a face in each object.