Summaries of video programs can be automatically created by identifying and compiling repeated portions of the program content. In many programs (such as broadcasts of sporting events), the most interesting portions of the event are often repeated more than once during the course of the program. By automatically identifying the replayed portions of the video presentation, then, a highlight summary of a television broadcast can be automatically created. Repeated imagery can be automatically identified by comparing metric values representing the contents of video frames. This technique can be implemented within a set top box or other television receiver that is operated by a viewer, thereby providing a high level of flexibility and convenience.