A system and method are disclosed for automatically generating a highlight reel of video content. Segments from a segment list may be associated with, or indexed to, corresponding sequences from a video of an event for which the segment list is prepared. Thereafter, also using the segment list, segments may be scored using a variety of predefined criteria to come up with segments which are likely to be of greatest interest to a particular user. The video sequences associated with the highest scored segments are used as the video highlight reel.