Implementations relate to an image cache used for replacing portions of images. In some implementations, a method includes examining a received image and detecting a subject depicted in a subject portion of the received image. The method determines a likelihood that the subject portion will be used as a replacement to be inserted in place of one or more corresponding subject portions in future images examined at a later time. The subject portion, or a reference thereto, is stored in an image cache in response to the determined likelihood being greater than a predetermined threshold. The image cache is available to be searched for the subject portion and other portions of images to replace corresponding portions of future images.