A distractor detector includes a heatmap network and a distractor classifier. The heatmap network operates on an input image to generate a heatmap for a main subject, a heatmap for a distractor, and optionally a heatmap for the background. Each object is cropped within the input image to generate a corresponding cropped image. Regions within the heatmaps that correspond to the objects are identified, and each of the regions is cropped within each of the heatmaps to generate cropped heatmaps. The distractor classifier then operates on the cropped images and the cropped heatmaps to classify each of the objects as being either a main subject or a distractor.